[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:44:32 dist-test-slave-s7k6 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:44:32 dist-test-slave-s7k6 krb5kdc[399](info): set up 2 sockets
Sep 26 14:44:32 dist-test-slave-s7k6 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 14:44:33.402855 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:44:33 dist-test-slave-s7k6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:44:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:44:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:44:33.522953 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40321
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:33.635269 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:33.635653 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:33.635727 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:33.640121 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:44:33.640190 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:33.640218 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:33.640239 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:44:33.640268 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:44:33.645895 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:33.647025 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:33.648113 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:33.657295 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:33.657428 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:33.657431 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:33.657625 412 server_base.cc:1034] running on GCE node
I20260926 14:44:33.658267 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:33.659539 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:33.660714 412 hybrid_clock.cc:648] HybridClock initialized: now 1790433873660665 us; error 65 us; skew 500 ppm
Sep 26 14:44:33 dist-test-slave-s7k6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:33.663657 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:44:33.664625 412 webserver.cc:460] Webserver started at http://127.0.98.62:37033/ using document root <none> and password file <none>
I20260926 14:44:33.665194 412 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:33.665263 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:33.665460 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:33.667747 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "7a77b2f183254cac89e805b07b769231"
format_stamp: "Formatted at 2026-09-26 14:44:33 on dist-test-slave-s7k6"
server_key: "4ba6b73dbf692206f3e63197cff60cb2"
server_key_iv: "d33fb13de4051f3f997d8350abed9351"
server_key_version: "encryptionkey@0"
I20260926 14:44:33.668365 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a77b2f183254cac89e805b07b769231"
format_stamp: "Formatted at 2026-09-26 14:44:33 on dist-test-slave-s7k6"
server_key: "4ba6b73dbf692206f3e63197cff60cb2"
server_key_iv: "d33fb13de4051f3f997d8350abed9351"
server_key_version: "encryptionkey@0"
I20260926 14:44:33.672300 412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:44:33.674968 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:33.676306 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:33.676436 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "7a77b2f183254cac89e805b07b769231"
format_stamp: "Formatted at 2026-09-26 14:44:33 on dist-test-slave-s7k6"
server_key: "4ba6b73dbf692206f3e63197cff60cb2"
server_key_iv: "d33fb13de4051f3f997d8350abed9351"
server_key_version: "encryptionkey@0"
I20260926 14:44:33.676611 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:33.693610 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:33.696686 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:33.696870 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:33.703657 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40321
I20260926 14:44:33.703717 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40321 every 8 connection(s)
I20260926 14:44:33.704656 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:44:33.707588 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:33.709317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 412
I20260926 14:44:33.709442 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:44:33.710702 392 external_mini_cluster.cc:1428] Setting key 618c9d179543082cd9cc1bbde5dc2698
I20260926 14:44:33.713657 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231: Bootstrap starting.
I20260926 14:44:33.716355 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:33.717115 480 log.cc:824] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:33.719031 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231: No bootstrap required, opened a new log
I20260926 14:44:33.722463 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77b2f183254cac89e805b07b769231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } }
I20260926 14:44:33.722700 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:33.722775 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a77b2f183254cac89e805b07b769231, State: Initialized, Role: FOLLOWER
I20260926 14:44:33.723322 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77b2f183254cac89e805b07b769231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } }
I20260926 14:44:33.723508 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:44:33.723577 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:44:33.723678 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:33.724795 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77b2f183254cac89e805b07b769231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } }
I20260926 14:44:33.725152 480 leader_election.cc:302] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a77b2f183254cac89e805b07b769231; no voters:
I20260926 14:44:33.725539 480 leader_election.cc:288] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:44:33.725759 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:33.726079 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [term 1 LEADER]: Becoming Leader. State: Replica: 7a77b2f183254cac89e805b07b769231, State: Running, Role: LEADER
I20260926 14:44:33.726428 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77b2f183254cac89e805b07b769231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } }
I20260926 14:44:33.726913 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:44:33.728185 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a77b2f183254cac89e805b07b769231. Latest consensus state: current_term: 1 leader_uuid: "7a77b2f183254cac89e805b07b769231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77b2f183254cac89e805b07b769231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } } }
I20260926 14:44:33.728312 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:33.728272 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a77b2f183254cac89e805b07b769231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a77b2f183254cac89e805b07b769231" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40321 } } }
I20260926 14:44:33.728366 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:33.728798 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 14:44:33 dist-test-slave-s7k6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:33.732138 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:44:33.736325 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:33.713853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44784 (local address 127.0.98.62:40321)
0926 14:44:33.714272 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:33.714287 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:33.715560 (+ 1273us) server_negotiation.cc:407] Connection header received
0926 14:44:33.718896 (+ 3336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:33.718914 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:33.719279 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:33.719602 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:33.721753 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:33.722707 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:33.724471 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:33.724963 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:33.725128 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:33.730133 (+ 5005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:33.730139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:33.730155 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:33.730188 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:33.732051 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:33.732670 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:33.732676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:33.732682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:33.732800 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:33.733149 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:33.733152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:33.733154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:33.733545 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:33.733716 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:33.735002 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0926 14:44:33.735316 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":112,"threads_started":1}
I20260926 14:44:33.739486 494 catalog_manager.cc:1347] Generated new cluster ID: 1fa5a90f830a42c787dbf520dfc73f35
I20260926 14:44:33.739562 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:44:33.752662 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:44:33.753656 494 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:44:33.765664 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a77b2f183254cac89e805b07b769231: Generated new TSK 0
I20260926 14:44:33.766427 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:44:33.797895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:33.919946 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:33.920217 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:33.920260 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:33.924274 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:33.924331 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:33.924391 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:44:33.929895 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:33.930962 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:33.931937 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:33.941599 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:33.941761 506 server_base.cc:1034] running on GCE node
W20260926 14:44:33.941742 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:33.941637 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:33.942349 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:33.943097 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:33.944267 506 hybrid_clock.cc:648] HybridClock initialized: now 1790433873944228 us; error 51 us; skew 500 ppm
Sep 26 14:44:33 dist-test-slave-s7k6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:33.947080 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:44:33.948057 506 webserver.cc:460] Webserver started at http://127.0.98.1:39933/ using document root <none> and password file <none>
I20260926 14:44:33.948623 506 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:33.948688 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:33.948885 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:33.951048 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "b75f8fbff4ae4803a6a7fe91322c686c"
format_stamp: "Formatted at 2026-09-26 14:44:33 on dist-test-slave-s7k6"
server_key: "71285a93e63a559edf5a95f7ab246df7"
server_key_iv: "e406702480cbdda665734a2eb7107bb0"
server_key_version: "encryptionkey@0"
I20260926 14:44:33.951628 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b75f8fbff4ae4803a6a7fe91322c686c"
format_stamp: "Formatted at 2026-09-26 14:44:33 on dist-test-slave-s7k6"
server_key: "71285a93e63a559edf5a95f7ab246df7"
server_key_iv: "e406702480cbdda665734a2eb7107bb0"
server_key_version: "encryptionkey@0"
I20260926 14:44:33.955478 506 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:44:33.958077 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:33.959472 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:44:33.959594 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "b75f8fbff4ae4803a6a7fe91322c686c"
format_stamp: "Formatted at 2026-09-26 14:44:33 on dist-test-slave-s7k6"
server_key: "71285a93e63a559edf5a95f7ab246df7"
server_key_iv: "e406702480cbdda665734a2eb7107bb0"
server_key_version: "encryptionkey@0"
I20260926 14:44:33.959695 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:33.980000 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:33.983450 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:33.983635 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:33.984272 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:33.985127 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:33.985175 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:33.985229 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:33.985257 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:33.994323 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34865
I20260926 14:44:33.994375 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34865 every 8 connection(s)
I20260926 14:44:33.995353 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:44:34.004751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 506
I20260926 14:44:34.004855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:44:34.005095 392 external_mini_cluster.cc:1428] Setting key 5b0270b9cc107fb4f570bfdd810e47dd
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:34.009766 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:33.997192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34673 (local address 127.0.98.62:40321)
0926 14:44:33.997374 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:33.997379 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:33.998334 (+ 955us) server_negotiation.cc:407] Connection header received
0926 14:44:33.999709 (+ 1375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:33.999713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:33.999761 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:33.999842 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:34.001136 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.001696 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.002911 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.003146 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.003228 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:34.005845 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:34.005848 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:34.005851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:34.005874 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:34.007582 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:34.008199 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:34.008203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:34.008205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:34.008307 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:34.008649 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:34.008652 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:34.008654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:34.008877 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:34.008969 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:34.009452 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 14:44:34.009600 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 14:44:34.010627 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:33.997659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40321 (local address 127.0.98.1:34673)
0926 14:44:33.998166 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:33.998217 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:33.999490 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:34.000010 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:34.000018 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:34.000390 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:34.000966 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:34.000980 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.001834 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.001837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:34.002805 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:34.002815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.003269 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.003272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:34.003422 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:34.004018 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:34.004036 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:34.005672 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:34.007706 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:34.007713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:34.007730 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:34.008077 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:34.008400 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:34.008402 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:34.008404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:34.008561 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:34.008968 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:34.008973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:34.009222 (+ 249us) client_negotiation.cc:769] Sending connection context
0926 14:44:34.009381 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 14:44:34.009706 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":118,"threads_started":1}
I20260926 14:44:34.011875 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40321
I20260926 14:44:34.012115 635 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:34.012598 635 heartbeater.cc:511] Master 127.0.98.62:40321 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 14:44:34.014175 444 ts_manager.cc:194] Registered new tserver with Master: b75f8fbff4ae4803a6a7fe91322c686c (127.0.98.1:34865)
I20260926 14:44:34.015400 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:44:34.031582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:34.145257 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:34.145550 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:34.145622 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:34.149793 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:34.149863 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:34.149943 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:44:34.155512 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:34.156622 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:34.157680 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:34.167409 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:34.167523 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:34.167645 640 server_base.cc:1034] running on GCE node
W20260926 14:44:34.167845 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:34.168273 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:34.168991 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:34.170171 640 hybrid_clock.cc:648] HybridClock initialized: now 1790433874170152 us; error 35 us; skew 500 ppm
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:34.173032 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:44:34.173952 640 webserver.cc:460] Webserver started at http://127.0.98.2:46433/ using document root <none> and password file <none>
I20260926 14:44:34.174568 640 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:34.174645 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:34.174885 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:34.177002 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "692c5175036a44b7ac92d90501cacd28"
format_stamp: "Formatted at 2026-09-26 14:44:34 on dist-test-slave-s7k6"
server_key: "50cd1fcc55834b38016dab281b386aa4"
server_key_iv: "1524055e8287ce5608b0ed16cc1b6a63"
server_key_version: "encryptionkey@0"
I20260926 14:44:34.177615 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "692c5175036a44b7ac92d90501cacd28"
format_stamp: "Formatted at 2026-09-26 14:44:34 on dist-test-slave-s7k6"
server_key: "50cd1fcc55834b38016dab281b386aa4"
server_key_iv: "1524055e8287ce5608b0ed16cc1b6a63"
server_key_version: "encryptionkey@0"
I20260926 14:44:34.182188 640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 14:44:34.185148 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:34.186568 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:44:34.186740 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "692c5175036a44b7ac92d90501cacd28"
format_stamp: "Formatted at 2026-09-26 14:44:34 on dist-test-slave-s7k6"
server_key: "50cd1fcc55834b38016dab281b386aa4"
server_key_iv: "1524055e8287ce5608b0ed16cc1b6a63"
server_key_version: "encryptionkey@0"
I20260926 14:44:34.186893 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:34.201825 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:34.204869 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:34.205034 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:34.205596 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:34.206389 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:34.206442 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:34.206483 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:34.206511 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:34.215696 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41171
I20260926 14:44:34.215735 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41171 every 8 connection(s)
I20260926 14:44:34.216616 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:44:34.217756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 640
I20260926 14:44:34.217873 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:44:34.218127 392 external_mini_cluster.cc:1428] Setting key 7ae735e67fa961122b4781023112408e
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:44:34.231348 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:34.218538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52845 (local address 127.0.98.62:40321)
0926 14:44:34.218665 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:34.218668 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:34.219822 (+ 1154us) server_negotiation.cc:407] Connection header received
0926 14:44:34.221567 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:34.221570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:34.221638 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:34.221714 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:34.223043 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.223636 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.224798 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.225016 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.225105 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:34.227608 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:34.227611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:34.227613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:34.227639 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:34.229190 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:34.229711 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:34.229715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:34.229717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:34.229792 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:34.230111 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:34.230114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:34.230116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:34.230343 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:34.230438 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:34.231038 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 14:44:34.231166 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:44:34.232136 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:34.219095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40321 (local address 127.0.98.2:52845)
0926 14:44:34.219648 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:34.219713 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:34.221313 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:34.221927 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:34.221935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:34.222317 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:34.222897 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:34.222914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.223758 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.223762 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:34.224677 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:34.224687 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.225169 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.225173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:34.225345 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:34.225920 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:34.225938 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:34.227425 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:34.229328 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:34.229334 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:34.229350 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:34.229604 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:34.229869 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:34.229872 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:34.229874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:34.230014 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:34.230463 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:34.230469 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:34.230748 (+ 279us) client_negotiation.cc:769] Sending connection context
0926 14:44:34.230961 (+ 213us) client_negotiation.cc:240] Negotiation successful
0926 14:44:34.231249 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":146,"threads_started":1}
I20260926 14:44:34.233788 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40321
I20260926 14:44:34.234030 769 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:34.234513 769 heartbeater.cc:511] Master 127.0.98.62:40321 requested a full tablet report, sending...
I20260926 14:44:34.236176 444 ts_manager.cc:194] Registered new tserver with Master: 692c5175036a44b7ac92d90501cacd28 (127.0.98.2:41171)
I20260926 14:44:34.236778 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:44:34.245576 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:34.362097 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:34.362373 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:34.362418 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:34.366827 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:34.366895 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:34.366963 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:44:34.372460 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40321
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:34.373613 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:34.374662 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:34.384851 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:34.384843 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:34.384843 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:34.385447 774 server_base.cc:1034] running on GCE node
I20260926 14:44:34.385946 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:34.386773 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:34.387949 774 hybrid_clock.cc:648] HybridClock initialized: now 1790433874387938 us; error 51 us; skew 500 ppm
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:34.391029 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:44:34.392005 774 webserver.cc:460] Webserver started at http://127.0.98.3:35895/ using document root <none> and password file <none>
I20260926 14:44:34.392606 774 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:34.392674 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:34.392864 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:34.395054 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "e3564280d24449149ec0a39f8c70ac01"
format_stamp: "Formatted at 2026-09-26 14:44:34 on dist-test-slave-s7k6"
server_key: "32e88d525d18c2c2d9a5ce185c448926"
server_key_iv: "e379ebccd509983bc7700aa9bc3d482f"
server_key_version: "encryptionkey@0"
I20260926 14:44:34.395661 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e3564280d24449149ec0a39f8c70ac01"
format_stamp: "Formatted at 2026-09-26 14:44:34 on dist-test-slave-s7k6"
server_key: "32e88d525d18c2c2d9a5ce185c448926"
server_key_iv: "e379ebccd509983bc7700aa9bc3d482f"
server_key_version: "encryptionkey@0"
I20260926 14:44:34.399725 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:44:34.402307 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:34.403720 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:44:34.403833 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "e3564280d24449149ec0a39f8c70ac01"
format_stamp: "Formatted at 2026-09-26 14:44:34 on dist-test-slave-s7k6"
server_key: "32e88d525d18c2c2d9a5ce185c448926"
server_key_iv: "e379ebccd509983bc7700aa9bc3d482f"
server_key_version: "encryptionkey@0"
I20260926 14:44:34.403934 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:34.416872 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:34.419965 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:34.420140 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:34.420732 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:34.421612 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:34.421659 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:34.421712 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:34.421736 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:34.431205 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34843
I20260926 14:44:34.431250 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34843 every 8 connection(s)
I20260926 14:44:34.432133 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:44:34.441874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 774
I20260926 14:44:34.441972 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:34.442245 392 external_mini_cluster.cc:1428] Setting key 18c2a7787732e8e8f38fe432766ea30c
I20260926 14:44:34.446607 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:34.433958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34423 (local address 127.0.98.62:40321)
0926 14:44:34.434125 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:34.434129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:34.435042 (+ 913us) server_negotiation.cc:407] Connection header received
0926 14:44:34.436624 (+ 1582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:34.436627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:34.436688 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:34.436764 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:34.438174 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.438749 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.439960 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.440188 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.440282 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:34.442849 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:34.442854 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:34.442856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:34.442888 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:34.444552 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:34.445077 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:34.445080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:34.445081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:34.445173 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:34.445511 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:34.445515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:34.445517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:34.445703 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:34.445791 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:34.446300 (+ 509us) server_negotiation.cc:299] Negotiation successful
0926 14:44:34.446441 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:44:34.447511 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:34.434387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40321 (local address 127.0.98.3:34423)
0926 14:44:34.434891 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:34.434936 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:34.436374 (+ 1438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:34.436976 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:34.436984 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:34.437347 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:34.437944 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:34.437959 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.438908 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.438911 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:34.439831 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:34.439842 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.440336 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.440340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:34.440523 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:34.441109 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:34.441127 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:34.442618 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:34.444676 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:34.444681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:34.444696 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:34.444963 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:34.445261 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:34.445264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:34.445266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:34.445419 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:34.445798 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:34.445805 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:34.446060 (+ 255us) client_negotiation.cc:769] Sending connection context
0926 14:44:34.446240 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 14:44:34.446587 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":112,"threads_started":1}
I20260926 14:44:34.448727 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40321
I20260926 14:44:34.448947 903 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:34.449437 903 heartbeater.cc:511] Master 127.0.98.62:40321 requested a full tablet report, sending...
I20260926 14:44:34.450469 444 ts_manager.cc:194] Registered new tserver with Master: e3564280d24449149ec0a39f8c70ac01 (127.0.98.3:34843)
I20260926 14:44:34.451073 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34423
I20260926 14:44:34.456643 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433874, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433874, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 14:44:34.483716 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:34.473641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.0.98.1:34865)
0926 14:44:34.473963 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:34.473983 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:34.474006 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:44:34.474056 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:34.474062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:34.474257 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:34.474385 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:34.475170 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.476020 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.476996 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.477407 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.477506 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:34.480011 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:34.480021 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:34.480034 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:34.480060 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:34.481822 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:34.482295 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:34.482301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:34.482303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:34.482389 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:34.482645 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:34.482671 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:34.482673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:34.482970 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:34.483195 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:34.483374 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 14:44:34.483544 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":125,"threads_started":1}
W20260926 14:44:34.484746 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53578
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 14:44:34.510686 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:34.502053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.0.98.1:34865)
0926 14:44:34.502227 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:34.502232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:34.502373 (+ 141us) server_negotiation.cc:407] Connection header received
0926 14:44:34.502516 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:34.502520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:34.502572 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:34.502648 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:34.503415 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.503944 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.504848 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:34.505098 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:34.505190 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:34.507655 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:34.507659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:34.507661 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:34.507689 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:34.509057 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:34.509509 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:34.509512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:34.509514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:34.509599 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:34.509867 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:34.509870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:34.509872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:34.510046 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:34.510170 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:34.510403 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 14:44:34.510527 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:44:34.511857 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 506
I20260926 14:44:34.517256 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 640
I20260926 14:44:34.524209 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 774
I20260926 14:44:34.529192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 412
2026-09-26T14:44:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2092 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[917](info): set up 2 sockets
Sep 26 14:44:34 dist-test-slave-s7k6 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 14:44:35.490298 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:44:35 dist-test-slave-s7k6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:44:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:44:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:44:35.606675 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36189
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:35.717945 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:35.718230 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:35.718290 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:35.722411 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:44:35.722479 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:35.722507 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:35.722528 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:44:35.722553 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:44:35.728397 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:35.729529 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:35.730654 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:44:35.739112 930 server_base.cc:1034] running on GCE node
W20260926 14:44:35.739073 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:35.739073 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:35.739087 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:35.739897 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:35.741062 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:35.742236 930 hybrid_clock.cc:648] HybridClock initialized: now 1790433875742217 us; error 33 us; skew 500 ppm
Sep 26 14:44:35 dist-test-slave-s7k6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:35.745307 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:44:35.746289 930 webserver.cc:460] Webserver started at http://127.0.98.62:39905/ using document root <none> and password file <none>
I20260926 14:44:35.746913 930 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:35.746991 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:35.747206 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:35.749307 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "b59c67436abc491bb3f49533c1eaff02"
format_stamp: "Formatted at 2026-09-26 14:44:35 on dist-test-slave-s7k6"
server_key: "63abe5145817509bd7b4dd2c0f5a715e"
server_key_iv: "7103611d176d9a22556a0d8af2e79272"
server_key_version: "encryptionkey@0"
I20260926 14:44:35.749868 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "b59c67436abc491bb3f49533c1eaff02"
format_stamp: "Formatted at 2026-09-26 14:44:35 on dist-test-slave-s7k6"
server_key: "63abe5145817509bd7b4dd2c0f5a715e"
server_key_iv: "7103611d176d9a22556a0d8af2e79272"
server_key_version: "encryptionkey@0"
I20260926 14:44:35.753762 930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:44:35.756249 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:35.757572 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:44:35.757719 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "b59c67436abc491bb3f49533c1eaff02"
format_stamp: "Formatted at 2026-09-26 14:44:35 on dist-test-slave-s7k6"
server_key: "63abe5145817509bd7b4dd2c0f5a715e"
server_key_iv: "7103611d176d9a22556a0d8af2e79272"
server_key_version: "encryptionkey@0"
I20260926 14:44:35.757881 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:35.780328 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:35.783532 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:35.783721 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:35.790987 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36189
I20260926 14:44:35.791083 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36189 every 8 connection(s)
I20260926 14:44:35.792028 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:44:35.792974 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 930
I20260926 14:44:35.793068 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:44:35.793319 392 external_mini_cluster.cc:1428] Setting key 4981cf3e723d7ab1fd9ef70625705b74
I20260926 14:44:35.795315 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:35.801261 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02: Bootstrap starting.
Sep 26 14:44:35 dist-test-slave-s7k6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:35.803797 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:35.804493 998 log.cc:824] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:35.806393 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02: No bootstrap required, opened a new log
I20260926 14:44:35.807399 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:35.794788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56652 (local address 127.0.98.62:36189)
0926 14:44:35.795249 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:35.795269 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:35.795309 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:44:35.795986 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:35.796005 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:35.796369 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:35.796705 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:35.797506 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:35.798570 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:35.799474 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:35.799982 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:35.800129 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:35.802826 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:35.802832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:35.802847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:35.802877 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:35.804605 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:35.805047 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:35.805052 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:35.805057 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:35.805173 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:35.805429 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:35.805433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:35.805434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:35.805845 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:35.806020 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:35.806307 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 14:44:35.806609 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":152,"threads_started":1}
I20260926 14:44:35.810472 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59c67436abc491bb3f49533c1eaff02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36189 } }
I20260926 14:44:35.810732 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:35.810817 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b59c67436abc491bb3f49533c1eaff02, State: Initialized, Role: FOLLOWER
I20260926 14:44:35.811282 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59c67436abc491bb3f49533c1eaff02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36189 } }
I20260926 14:44:35.811442 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:44:35.811513 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:44:35.811615 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:35.812609 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59c67436abc491bb3f49533c1eaff02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36189 } }
I20260926 14:44:35.812928 998 leader_election.cc:302] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b59c67436abc491bb3f49533c1eaff02; no voters:
I20260926 14:44:35.813197 998 leader_election.cc:288] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:44:35.813341 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:35.813561 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [term 1 LEADER]: Becoming Leader. State: Replica: b59c67436abc491bb3f49533c1eaff02, State: Running, Role: LEADER
I20260926 14:44:35.813937 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59c67436abc491bb3f49533c1eaff02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36189 } }
I20260926 14:44:35.814401 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:44:35.815529 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b59c67436abc491bb3f49533c1eaff02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59c67436abc491bb3f49533c1eaff02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36189 } } }
I20260926 14:44:35.815645 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:35.815977 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b59c67436abc491bb3f49533c1eaff02. Latest consensus state: current_term: 1 leader_uuid: "b59c67436abc491bb3f49533c1eaff02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b59c67436abc491bb3f49533c1eaff02" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36189 } } }
I20260926 14:44:35.816124 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:35.816438 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:44:35.819422 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:44:35.824936 1012 catalog_manager.cc:1347] Generated new cluster ID: f9824f53c1f7404aa49721ebc1928320
I20260926 14:44:35.825016 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:44:35.841753 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:44:35.843098 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:44:35.852521 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b59c67436abc491bb3f49533c1eaff02: Generated new TSK 0
I20260926 14:44:35.853428 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:44:35.881888 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36189
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:35.993397 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:35.993677 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:35.993724 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:35.997861 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:35.997947 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:35.998011 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:44:36.003501 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36189
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:36.004570 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:36.005628 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:36.015563 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:36.015759 1024 server_base.cc:1034] running on GCE node
W20260926 14:44:36.015568 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:36.015568 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:36.016328 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:36.017019 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:36.018201 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790433876018168 us; error 46 us; skew 500 ppm
Sep 26 14:44:36 dist-test-slave-s7k6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:36.021200 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:44:36.022261 1024 webserver.cc:460] Webserver started at http://127.0.98.1:39157/ using document root <none> and password file <none>
I20260926 14:44:36.022903 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:36.022990 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:36.023205 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:36.025336 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "b6a3d6df5d61497bbf07d0d636806d97"
format_stamp: "Formatted at 2026-09-26 14:44:36 on dist-test-slave-s7k6"
server_key: "4428917b19a7c2bcde93d870ef83c1e6"
server_key_iv: "c4864dfa5a5c583cf33a08b6a375a089"
server_key_version: "encryptionkey@0"
I20260926 14:44:36.025929 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b6a3d6df5d61497bbf07d0d636806d97"
format_stamp: "Formatted at 2026-09-26 14:44:36 on dist-test-slave-s7k6"
server_key: "4428917b19a7c2bcde93d870ef83c1e6"
server_key_iv: "c4864dfa5a5c583cf33a08b6a375a089"
server_key_version: "encryptionkey@0"
I20260926 14:44:36.029974 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:44:36.032521 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:36.033759 1024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:44:36.033875 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "b6a3d6df5d61497bbf07d0d636806d97"
format_stamp: "Formatted at 2026-09-26 14:44:36 on dist-test-slave-s7k6"
server_key: "4428917b19a7c2bcde93d870ef83c1e6"
server_key_iv: "c4864dfa5a5c583cf33a08b6a375a089"
server_key_version: "encryptionkey@0"
I20260926 14:44:36.033995 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:36.048172 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:36.051657 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:36.051893 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:36.052485 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:36.053339 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:36.053385 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:36.053428 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:36.053457 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:36.061658 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43843
I20260926 14:44:36.061697 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43843 every 8 connection(s)
I20260926 14:44:36.062631 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:44:36.067938 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 1024
I20260926 14:44:36.068048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:44:36.068341 392 external_mini_cluster.cc:1428] Setting key 6e02bb51338de896f4b9f25ac5a9ebcc
Sep 26 14:44:36 dist-test-slave-s7k6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 14:44:36.078580 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:36.064495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43523 (local address 127.0.98.62:36189)
0926 14:44:36.064621 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:36.064625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:36.065584 (+ 959us) server_negotiation.cc:407] Connection header received
0926 14:44:36.067020 (+ 1436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:36.067023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:36.067085 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:36.067160 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:36.068745 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:36.069550 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:36.070791 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:36.071038 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:36.071128 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:36.074303 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:36.074306 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:36.074309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:36.074336 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:36.075905 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:36.076588 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:36.076591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:36.076592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:36.076680 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:36.077092 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:36.077094 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:36.077095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:36.077281 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:36.077358 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:36.078078 (+ 720us) server_negotiation.cc:299] Negotiation successful
0926 14:44:36.078429 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:44:36.079195 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:36.064919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36189 (local address 127.0.98.1:43523)
0926 14:44:36.065417 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:36.065461 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:36.066780 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:36.067373 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:36.067380 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:36.067796 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:36.068524 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:36.068550 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:36.069680 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:36.069684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:36.070626 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:36.070638 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:36.071180 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:36.071184 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:36.071414 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:36.072144 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:36.072166 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:36.074016 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:36.076037 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:36.076046 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:36.076067 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:36.076453 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:36.076780 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:36.076783 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:36.076785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:36.076998 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:36.077379 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:36.077386 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:36.077781 (+ 395us) client_negotiation.cc:769] Sending connection context
0926 14:44:36.077989 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 14:44:36.078328 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":116,"threads_started":1}
I20260926 14:44:36.080502 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36189
I20260926 14:44:36.080762 1153 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:36.081300 1153 heartbeater.cc:511] Master 127.0.98.62:36189 requested a full tablet report, sending...
I20260926 14:44:36.082950 962 ts_manager.cc:194] Registered new tserver with Master: b6a3d6df5d61497bbf07d0d636806d97 (127.0.98.1:43843)
I20260926 14:44:36.084218 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
W20260926 14:44:37.034994 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.2: real 0.965s user 0.000s sys 0.003s
I20260926 14:44:37.035475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36189
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:44:37.086743 1153 heartbeater.cc:503] Master 127.0.98.62:36189 was elected leader, sending a full tablet report...
W20260926 14:44:37.146054 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:37.146342 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:37.146386 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:37.150550 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:37.150619 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:37.150681 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:44:37.156189 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36189
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:37.157271 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:37.158296 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:37.167819 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:37.167820 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:37.167819 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:37.168175 1158 server_base.cc:1034] running on GCE node
I20260926 14:44:37.168566 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:37.169235 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:37.170398 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790433877170377 us; error 35 us; skew 500 ppm
Sep 26 14:44:37 dist-test-slave-s7k6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:37.173307 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:44:37.174228 1158 webserver.cc:460] Webserver started at http://127.0.98.2:32901/ using document root <none> and password file <none>
I20260926 14:44:37.174830 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:37.174902 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:37.175103 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:37.177229 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "2e90d83d640a4ed485220935c129fd3c"
format_stamp: "Formatted at 2026-09-26 14:44:37 on dist-test-slave-s7k6"
server_key: "c2c245e04a2bdeae8fcd791f81eafe24"
server_key_iv: "2a8c9dd0ceb87c1d8f9c42710858aec1"
server_key_version: "encryptionkey@0"
I20260926 14:44:37.177804 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2e90d83d640a4ed485220935c129fd3c"
format_stamp: "Formatted at 2026-09-26 14:44:37 on dist-test-slave-s7k6"
server_key: "c2c245e04a2bdeae8fcd791f81eafe24"
server_key_iv: "2a8c9dd0ceb87c1d8f9c42710858aec1"
server_key_version: "encryptionkey@0"
I20260926 14:44:37.181802 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 14:44:37.185070 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:37.186395 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 14:44:37.186513 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "2e90d83d640a4ed485220935c129fd3c"
format_stamp: "Formatted at 2026-09-26 14:44:37 on dist-test-slave-s7k6"
server_key: "c2c245e04a2bdeae8fcd791f81eafe24"
server_key_iv: "2a8c9dd0ceb87c1d8f9c42710858aec1"
server_key_version: "encryptionkey@0"
I20260926 14:44:37.186621 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:37.202440 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:37.205808 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:37.206003 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:37.206617 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:37.207490 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:37.207541 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:37.207593 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:37.207623 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:37.217242 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38983
I20260926 14:44:37.217273 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38983 every 8 connection(s)
I20260926 14:44:37.218210 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:44:37.221776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 1158
I20260926 14:44:37.221892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:44:37.222138 392 external_mini_cluster.cc:1428] Setting key e8e86fca6001f484a5e75335abc0d40e
Sep 26 14:44:37 dist-test-slave-s7k6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 14:44:37.234659 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:37.220180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60449 (local address 127.0.98.62:36189)
0926 14:44:37.220499 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:37.220503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:37.221331 (+ 828us) server_negotiation.cc:407] Connection header received
0926 14:44:37.222995 (+ 1664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:37.222998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:37.223050 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:37.223153 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:37.224831 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:37.225656 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:37.227246 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:37.227484 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:37.227579 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:37.230574 (+ 2995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:37.230583 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:37.230585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:37.230612 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:37.232395 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:37.232946 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:37.232992 (+ 46us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:37.232994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:37.233088 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:37.233490 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:37.233493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:37.233494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:37.233698 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:37.233797 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:37.234391 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 14:44:37.234509 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":105,"threads_started":1}
I20260926 14:44:37.235479 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:37.220631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36189 (local address 127.0.98.2:60449)
0926 14:44:37.221153 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:37.221204 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:37.222678 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:37.223342 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:37.223351 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:37.223899 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:37.224651 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:37.224673 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:37.225808 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:37.225812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:37.227105 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:37.227115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:37.227635 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:37.227639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:37.227809 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:37.228456 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:37.228480 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:37.230352 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:37.232519 (+ 2167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:37.232525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:37.232540 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:37.232830 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:37.233215 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:37.233218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:37.233219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:37.233373 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:37.233809 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:37.233814 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:37.234143 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 14:44:37.234310 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 14:44:37.234599 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":132,"threads_started":1}
I20260926 14:44:37.236755 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36189
I20260926 14:44:37.237061 1287 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:37.237571 1287 heartbeater.cc:511] Master 127.0.98.62:36189 requested a full tablet report, sending...
I20260926 14:44:37.238747 962 ts_manager.cc:194] Registered new tserver with Master: 2e90d83d640a4ed485220935c129fd3c (127.0.98.2:38983)
I20260926 14:44:37.239343 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:44:38.241469 1287 heartbeater.cc:503] Master 127.0.98.62:36189 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260926 14:44:38.412290 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 1.188s user 0.003s sys 0.000s
I20260926 14:44:38.412801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36189
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:38.531759 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:38.532063 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:38.532109 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:38.536532 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:38.536755 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:38.536857 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:44:38.543131 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36189
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:38.544425 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:38.545630 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:38.555701 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:38.555786 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:38.555696 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:38.556144 1293 server_base.cc:1034] running on GCE node
I20260926 14:44:38.556665 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:38.557392 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:38.558562 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790433878558540 us; error 38 us; skew 500 ppm
Sep 26 14:44:38 dist-test-slave-s7k6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:38.571513 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:44:38.572686 1293 webserver.cc:460] Webserver started at http://127.0.98.3:42129/ using document root <none> and password file <none>
I20260926 14:44:38.573316 1293 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:38.573391 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:38.573597 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:38.575949 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "6d0d2024bc324e4da1981140e2c0c518"
format_stamp: "Formatted at 2026-09-26 14:44:38 on dist-test-slave-s7k6"
server_key: "8553ea5415148d9c6d7aa9aaf8ea85b4"
server_key_iv: "90205c7887cb787e8ba3c925c4e604ca"
server_key_version: "encryptionkey@0"
I20260926 14:44:38.576573 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6d0d2024bc324e4da1981140e2c0c518"
format_stamp: "Formatted at 2026-09-26 14:44:38 on dist-test-slave-s7k6"
server_key: "8553ea5415148d9c6d7aa9aaf8ea85b4"
server_key_iv: "90205c7887cb787e8ba3c925c4e604ca"
server_key_version: "encryptionkey@0"
I20260926 14:44:38.580770 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 14:44:38.583642 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:38.585178 1293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:44:38.585326 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "6d0d2024bc324e4da1981140e2c0c518"
format_stamp: "Formatted at 2026-09-26 14:44:38 on dist-test-slave-s7k6"
server_key: "8553ea5415148d9c6d7aa9aaf8ea85b4"
server_key_iv: "90205c7887cb787e8ba3c925c4e604ca"
server_key_version: "encryptionkey@0"
I20260926 14:44:38.585457 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:38.625200 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:38.629124 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:38.629344 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:38.630060 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:38.631073 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:38.631135 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:38.631207 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:38.631233 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:38.640144 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42003
I20260926 14:44:38.640175 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42003 every 8 connection(s)
I20260926 14:44:38.641079 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:44:38.650033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 1293
I20260926 14:44:38.650147 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:44:38.650393 392 external_mini_cluster.cc:1428] Setting key af79c07e3f3ea7b647508380d2c0af9e
Sep 26 14:44:38 dist-test-slave-s7k6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:38.656260 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.643034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53571 (local address 127.0.98.62:36189)
0926 14:44:38.643318 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:38.643322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:38.644072 (+ 750us) server_negotiation.cc:407] Connection header received
0926 14:44:38.645506 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:38.645509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:38.645575 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:38.645686 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:38.647032 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.647645 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.648849 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.649105 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.649197 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.652503 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:38.652507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:38.652511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:38.652541 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:38.654286 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:38.654850 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:38.654853 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:38.654855 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:38.654935 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:38.655215 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:38.655217 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:38.655218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:38.655414 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:38.655486 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:38.655944 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 14:44:38.656072 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":104,"threads_started":1}
I20260926 14:44:38.657109 1425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.643442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36189 (local address 127.0.98.3:53571)
0926 14:44:38.643909 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:38.643957 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:38.645264 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:38.645886 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:38.645893 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:38.646297 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:38.646873 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.646890 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.647774 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.647777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.648711 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.648721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.649272 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.649276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.649448 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.650038 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:38.650056 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:38.652271 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:38.654422 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:38.654428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:38.654442 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:38.654731 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:38.654997 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:38.655000 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:38.655001 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:38.655153 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:38.655487 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:38.655492 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:38.655744 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 14:44:38.655899 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 14:44:38.656249 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":121,"threads_started":1}
I20260926 14:44:38.658275 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36189
I20260926 14:44:38.658514 1422 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:38.659044 1422 heartbeater.cc:511] Master 127.0.98.62:36189 requested a full tablet report, sending...
I20260926 14:44:38.660087 962 ts_manager.cc:194] Registered new tserver with Master: 6d0d2024bc324e4da1981140e2c0c518 (127.0.98.3:42003)
I20260926 14:44:38.660670 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53571
I20260926 14:44:38.664570 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:44:38.675942 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.668229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.0.98.62:36189)
0926 14:44:38.668397 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:38.668402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:38.668503 (+ 101us) server_negotiation.cc:407] Connection header received
0926 14:44:38.668656 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:38.668659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:38.668704 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:38.668781 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:38.669540 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.670058 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.670984 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.671260 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.671343 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.672778 (+ 1435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:38.672781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:38.672783 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:38.672812 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:38.674295 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:38.674722 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:38.674725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:38.674726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:38.674855 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:38.675086 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:38.675088 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:38.675089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:38.675369 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:38.675478 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:38.675662 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 14:44:38.675780 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 14:44:38.681430 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56654:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:44:38.684903 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:44:38.700502 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.695107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37536 (local address 127.0.98.3:42003)
0926 14:44:38.696160 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:38.696227 (+ 67us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:38.696250 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:44:38.696339 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:38.696345 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:38.696534 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:38.696812 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:38.697040 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.697819 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.699013 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.699758 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.699860 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.700019 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:38.700240 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 14:44:38.700342 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":949,"thread_start_us":116,"threads_started":1}
I20260926 14:44:38.700497 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.694039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38983 (local address 127.0.0.1:53372)
0926 14:44:38.694480 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:38.694534 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:38.694680 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:38.695509 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:38.695515 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:38.695538 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:38.695711 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.695716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.697210 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.697213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.698694 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.698704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.699897 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.699901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.700044 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.700089 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 14:44:38.700184 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 14:44:38.700339 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":102,"thread_start_us":112,"threads_started":1}
I20260926 14:44:38.700527 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.694983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55020 (local address 127.0.98.1:43843)
0926 14:44:38.695237 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:38.695244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:38.695268 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:44:38.695347 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:38.695353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:38.695554 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:38.695690 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:38.695894 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.696957 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.698876 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.699757 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.699946 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.700095 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:38.700308 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 14:44:38.700407 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260926 14:44:38.700654 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.694879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42003 (local address 127.0.0.1:37536)
0926 14:44:38.695327 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:38.695340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:38.695419 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:38.696796 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:38.696799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:38.696818 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:38.696920 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.696925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.697962 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.697965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.698862 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.698873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.699874 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.699878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.700040 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.700085 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 14:44:38.700210 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 14:44:38.700560 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":320,"thread_start_us":84,"threads_started":1}
I20260926 14:44:38.700500 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.694110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43843 (local address 127.0.0.1:55020)
0926 14:44:38.694556 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:38.694571 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:38.694680 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:38.695671 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:38.695674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:38.695690 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:38.695803 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.695810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.697075 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.697078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.698694 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.698704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.699897 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.699901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.700153 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.700171 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:44:38.700229 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:44:38.700347 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":264,"thread_start_us":112,"threads_started":1}
I20260926 14:44:38.701381 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.694175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.0.98.2:38983)
0926 14:44:38.695000 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:38.695008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:38.695032 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:44:38.695110 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:38.695115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:38.695347 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:38.695862 (+ 515us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:38.696004 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.697100 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.698875 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.699760 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.700981 (+ 1221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.701100 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:38.701150 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 14:44:38.701247 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":110,"threads_started":1}
I20260926 14:44:38.704085 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 38bba8e2038b4061a72570b67d5ff007 (DEFAULT_TABLE table=test-table [id=c7badc55a7a54f969d6d4cbf74fd0fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:38.704101 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 38bba8e2038b4061a72570b67d5ff007 (DEFAULT_TABLE table=test-table [id=c7badc55a7a54f969d6d4cbf74fd0fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:38.704101 1356 tablet_service.cc:1467] Processing CreateTablet for tablet 38bba8e2038b4061a72570b67d5ff007 (DEFAULT_TABLE table=test-table [id=c7badc55a7a54f969d6d4cbf74fd0fdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:38.705238 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38bba8e2038b4061a72570b67d5ff007. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:38.705240 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38bba8e2038b4061a72570b67d5ff007. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:38.705240 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38bba8e2038b4061a72570b67d5ff007. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:38.712196 1441 tablet_bootstrap.cc:492] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c: Bootstrap starting.
I20260926 14:44:38.713665 1442 tablet_bootstrap.cc:492] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97: Bootstrap starting.
I20260926 14:44:38.713888 1443 tablet_bootstrap.cc:492] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518: Bootstrap starting.
I20260926 14:44:38.714795 1441 tablet_bootstrap.cc:654] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:38.715694 1441 log.cc:824] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:38.716018 1443 tablet_bootstrap.cc:654] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:38.716018 1442 tablet_bootstrap.cc:654] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:38.716907 1442 log.cc:824] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:38.717917 1441 tablet_bootstrap.cc:492] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c: No bootstrap required, opened a new log
I20260926 14:44:38.717875 1443 log.cc:824] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:38.718142 1441 ts_tablet_manager.cc:1397] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:44:38.719136 1442 tablet_bootstrap.cc:492] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97: No bootstrap required, opened a new log
I20260926 14:44:38.719337 1442 ts_tablet_manager.cc:1397] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:44:38.719954 1443 tablet_bootstrap.cc:492] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518: No bootstrap required, opened a new log
I20260926 14:44:38.720149 1443 ts_tablet_manager.cc:1397] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:44:38.721946 1441 raft_consensus.cc:348] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e90d83d640a4ed485220935c129fd3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "b6a3d6df5d61497bbf07d0d636806d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } } peers { permanent_uuid: "6d0d2024bc324e4da1981140e2c0c518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42003 } }
I20260926 14:44:38.722183 1441 raft_consensus.cc:374] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:38.722265 1441 raft_consensus.cc:729] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e90d83d640a4ed485220935c129fd3c, State: Initialized, Role: FOLLOWER
I20260926 14:44:38.722710 1441 consensus_queue.cc:260] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e90d83d640a4ed485220935c129fd3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "b6a3d6df5d61497bbf07d0d636806d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } } peers { permanent_uuid: "6d0d2024bc324e4da1981140e2c0c518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42003 } }
I20260926 14:44:38.723819 1441 ts_tablet_manager.cc:1428] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:44:38.724112 1443 raft_consensus.cc:348] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e90d83d640a4ed485220935c129fd3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "b6a3d6df5d61497bbf07d0d636806d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } } peers { permanent_uuid: "6d0d2024bc324e4da1981140e2c0c518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42003 } }
I20260926 14:44:38.724486 1443 raft_consensus.cc:374] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:38.724575 1443 raft_consensus.cc:729] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d0d2024bc324e4da1981140e2c0c518, State: Initialized, Role: FOLLOWER
I20260926 14:44:38.725117 1443 consensus_queue.cc:260] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e90d83d640a4ed485220935c129fd3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "b6a3d6df5d61497bbf07d0d636806d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } } peers { permanent_uuid: "6d0d2024bc324e4da1981140e2c0c518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42003 } }
I20260926 14:44:38.725270 1442 raft_consensus.cc:348] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e90d83d640a4ed485220935c129fd3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "b6a3d6df5d61497bbf07d0d636806d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } } peers { permanent_uuid: "6d0d2024bc324e4da1981140e2c0c518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42003 } }
I20260926 14:44:38.725594 1442 raft_consensus.cc:374] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:38.725694 1442 raft_consensus.cc:729] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6a3d6df5d61497bbf07d0d636806d97, State: Initialized, Role: FOLLOWER
I20260926 14:44:38.725989 1422 heartbeater.cc:503] Master 127.0.98.62:36189 was elected leader, sending a full tablet report...
I20260926 14:44:38.726241 1443 ts_tablet_manager.cc:1428] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260926 14:44:38.726183 1442 consensus_queue.cc:260] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e90d83d640a4ed485220935c129fd3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "b6a3d6df5d61497bbf07d0d636806d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } } peers { permanent_uuid: "6d0d2024bc324e4da1981140e2c0c518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42003 } }
I20260926 14:44:38.727182 1442 ts_tablet_manager.cc:1428] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260926 14:44:38.815613 1154 tablet.cc:2369] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:44:38.870606 1447 raft_consensus.cc:482] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:44:38.870911 1447 raft_consensus.cc:504] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e90d83d640a4ed485220935c129fd3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "b6a3d6df5d61497bbf07d0d636806d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } } peers { permanent_uuid: "6d0d2024bc324e4da1981140e2c0c518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42003 } }
I20260926 14:44:38.872225 1447 leader_election.cc:288] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6a3d6df5d61497bbf07d0d636806d97 (127.0.98.1:43843), 6d0d2024bc324e4da1981140e2c0c518 (127.0.98.3:42003)
I20260926 14:44:38.876811 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.872668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44623 (local address 127.0.98.1:43843)
0926 14:44:38.872858 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:38.872862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:38.873048 (+ 186us) server_negotiation.cc:407] Connection header received
0926 14:44:38.873275 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:38.873279 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:38.873339 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:38.873431 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:38.873756 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.874362 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.875718 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.876189 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.876278 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.876341 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:38.876580 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 14:44:38.876668 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 14:44:38.876832 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.873038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42003 (local address 127.0.98.2:41623)
0926 14:44:38.873377 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:38.873390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:38.873478 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:38.873756 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:38.873759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:38.873777 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:38.873896 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.873902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.874819 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.874822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.875680 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.875694 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.876408 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.876411 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.876490 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.876508 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:44:38.876580 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 14:44:38.876695 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":27,"thread_start_us":108,"threads_started":1}
I20260926 14:44:38.876878 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.873273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41623 (local address 127.0.98.3:42003)
0926 14:44:38.873391 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:38.873394 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:38.873445 (+ 51us) server_negotiation.cc:407] Connection header received
0926 14:44:38.873577 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:38.873579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:38.873625 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:38.873715 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:38.874043 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.874645 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.875835 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.876280 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.876393 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.876452 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:38.876607 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 14:44:38.876679 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:44:38.876835 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.872588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43843 (local address 127.0.98.2:44623)
0926 14:44:38.872948 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:38.872960 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:38.873093 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:38.873465 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:38.873469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:38.873493 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:38.873645 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.873652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.874504 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.874508 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.875546 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:38.875560 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.876336 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.876340 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:38.876445 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.876470 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:44:38.876541 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:44:38.876679 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":1,"thread_start_us":139,"threads_started":1}
I20260926 14:44:38.877535 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bba8e2038b4061a72570b67d5ff007" candidate_uuid: "2e90d83d640a4ed485220935c129fd3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d0d2024bc324e4da1981140e2c0c518" is_pre_election: true
I20260926 14:44:38.877666 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bba8e2038b4061a72570b67d5ff007" candidate_uuid: "2e90d83d640a4ed485220935c129fd3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a3d6df5d61497bbf07d0d636806d97" is_pre_election: true
I20260926 14:44:38.877913 1107 raft_consensus.cc:2454] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e90d83d640a4ed485220935c129fd3c in term 0.
I20260926 14:44:38.877898 1376 raft_consensus.cc:2454] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e90d83d640a4ed485220935c129fd3c in term 0.
I20260926 14:44:38.878459 1177 leader_election.cc:302] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e90d83d640a4ed485220935c129fd3c, b6a3d6df5d61497bbf07d0d636806d97; no voters:
I20260926 14:44:38.878713 1447 raft_consensus.cc:2781] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:44:38.878834 1447 raft_consensus.cc:482] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:44:38.878888 1447 raft_consensus.cc:3037] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [term 0 FOLLOWER]: Advancing to term 1
W20260926 14:44:38.891975 1423 tablet.cc:2369] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:44:38.921136 1447 raft_consensus.cc:504] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e90d83d640a4ed485220935c129fd3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "b6a3d6df5d61497bbf07d0d636806d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } } peers { permanent_uuid: "6d0d2024bc324e4da1981140e2c0c518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42003 } }
I20260926 14:44:38.921712 1447 leader_election.cc:288] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [CANDIDATE]: Term 1 election: Requested vote from peers b6a3d6df5d61497bbf07d0d636806d97 (127.0.98.1:43843), 6d0d2024bc324e4da1981140e2c0c518 (127.0.98.3:42003)
I20260926 14:44:38.922354 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bba8e2038b4061a72570b67d5ff007" candidate_uuid: "2e90d83d640a4ed485220935c129fd3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d0d2024bc324e4da1981140e2c0c518"
I20260926 14:44:38.922257 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bba8e2038b4061a72570b67d5ff007" candidate_uuid: "2e90d83d640a4ed485220935c129fd3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6a3d6df5d61497bbf07d0d636806d97"
I20260926 14:44:38.922528 1376 raft_consensus.cc:3037] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:38.922680 1107 raft_consensus.cc:3037] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:38.924103 1376 raft_consensus.cc:2454] T 38bba8e2038b4061a72570b67d5ff007 P 6d0d2024bc324e4da1981140e2c0c518 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e90d83d640a4ed485220935c129fd3c in term 1.
I20260926 14:44:38.924103 1107 raft_consensus.cc:2454] T 38bba8e2038b4061a72570b67d5ff007 P b6a3d6df5d61497bbf07d0d636806d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e90d83d640a4ed485220935c129fd3c in term 1.
I20260926 14:44:38.924539 1177 leader_election.cc:302] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e90d83d640a4ed485220935c129fd3c, 6d0d2024bc324e4da1981140e2c0c518; no voters:
I20260926 14:44:38.924801 1447 raft_consensus.cc:2781] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:38.925055 1447 raft_consensus.cc:686] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [term 1 LEADER]: Becoming Leader. State: Replica: 2e90d83d640a4ed485220935c129fd3c, State: Running, Role: LEADER
I20260926 14:44:38.925462 1447 consensus_queue.cc:237] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e90d83d640a4ed485220935c129fd3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } } peers { permanent_uuid: "b6a3d6df5d61497bbf07d0d636806d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } } peers { permanent_uuid: "6d0d2024bc324e4da1981140e2c0c518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42003 } }
I20260926 14:44:38.928889 960 catalog_manager.cc:5495] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e90d83d640a4ed485220935c129fd3c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2e90d83d640a4ed485220935c129fd3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e90d83d640a4ed485220935c129fd3c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6a3d6df5d61497bbf07d0d636806d97" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d0d2024bc324e4da1981140e2c0c518" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42003 } health_report { overall_health: UNKNOWN } } }
W20260926 14:44:38.970543 1288 tablet.cc:2369] T 38bba8e2038b4061a72570b67d5ff007 P 2e90d83d640a4ed485220935c129fd3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 14:44:38 dist-test-slave-s7k6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433875, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 14:44:38.997704 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:38.987150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.0.98.1:43843)
0926 14:44:38.987335 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:38.987338 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:38.987547 (+ 209us) server_negotiation.cc:407] Connection header received
0926 14:44:38.987716 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:38.987719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:38.987766 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:38.987845 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:38.988707 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.989361 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.990562 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:38.990845 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:38.990984 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:38.993642 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:38.993657 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:38.993673 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:38.993703 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:38.995638 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:38.996179 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:38.996185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:38.996187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:38.996342 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:38.996667 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:38.996669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:38.996670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:38.996965 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:38.997136 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:38.997436 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 14:44:38.997549 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 14:44:39 dist-test-slave-s7k6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433879, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:44:39 dist-test-slave-s7k6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433879, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 14:44:39.075203 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:39.065824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55034 (local address 127.0.98.1:43843)
0926 14:44:39.065954 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:39.065958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:39.066257 (+ 299us) server_negotiation.cc:407] Connection header received
0926 14:44:39.066391 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:39.066394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:39.066452 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:39.066538 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:39.067597 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:39.068408 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:39.069244 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:39.069551 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:39.069648 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:39.072033 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:39.072037 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:39.072039 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:39.072064 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:39.073500 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:39.073924 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:39.073927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:39.073928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:39.074018 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:39.074333 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:39.074336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:39.074337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:39.074522 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:39.074638 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:39.074913 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 14:44:39.075039 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 14:44:39 dist-test-slave-s7k6 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:44:39 dist-test-slave-s7k6 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433879, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 14:44:39.175925 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:39.166784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55042 (local address 127.0.98.1:43843)
0926 14:44:39.166907 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:39.166912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:39.167151 (+ 239us) server_negotiation.cc:407] Connection header received
0926 14:44:39.167274 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:39.167277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:39.167326 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:39.167393 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:39.168165 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:39.168706 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:39.169652 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:39.169885 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:39.169978 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:39.172528 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:39.172531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:39.172533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:39.172565 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:39.174086 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:39.174617 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:39.174622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:39.174623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:39.174712 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:39.175020 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:39.175023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:39.175024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:39.175219 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:39.175330 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:39.175655 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 14:44:39.175777 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 14:44:39.177717 1067 tablet_service.cc:2908] Rejecting scan request for tablet 38bba8e2038b4061a72570b67d5ff007: Uninitialized: clean time has not yet been initialized
I20260926 14:44:39.179948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 1024
I20260926 14:44:39.186110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 1158
I20260926 14:44:39.191452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 1293
I20260926 14:44:39.196578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 930
2026-09-26T14:44:39Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4669 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:44:40 dist-test-slave-s7k6 krb5kdc[1466](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:44:40 dist-test-slave-s7k6 krb5kdc[1466](info): set up 2 sockets
Sep 26 14:44:40 dist-test-slave-s7k6 krb5kdc[1466](info): commencing operation
krb5kdc: starting...
W20260926 14:44:41.322665 392 mini_kdc.cc:121] Time spent starting KDC: real 2.103s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 14:44:41.482956 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.160s user 0.000s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 14:44:41.759778 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.277s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:44:41 dist-test-slave-s7k6 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:44:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:44:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:44:41.836913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44975
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:41.947359 1479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:41.947670 1479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:41.947726 1479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:41.952039 1479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:44:41.952112 1479 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:44:41.952136 1479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:41.952159 1479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:41.952179 1479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:44:41.952208 1479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:44:41.957794 1479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44975
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:41.958983 1479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:41.960057 1479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:41.968607 1484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:41.968641 1487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:41.968614 1485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:41.968799 1479 server_base.cc:1034] running on GCE node
I20260926 14:44:41.969460 1479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:41.970561 1479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:41.971725 1479 hybrid_clock.cc:648] HybridClock initialized: now 1790433881971716 us; error 48 us; skew 500 ppm
Sep 26 14:44:41 dist-test-slave-s7k6 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:41.974522 1479 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:44:41.975561 1479 webserver.cc:460] Webserver started at http://127.0.98.62:39147/ using document root <none> and password file <none>
I20260926 14:44:41.976128 1479 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:41.976197 1479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:41.976408 1479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:41.978581 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "90a530c4d9bb4117ac3ab6b1529cb250"
format_stamp: "Formatted at 2026-09-26 14:44:41 on dist-test-slave-s7k6"
server_key: "df6822b70acf56c5c98d0aa32e08269d"
server_key_iv: "1aa5f9a422c54112ba835592d21840b4"
server_key_version: "encryptionkey@0"
I20260926 14:44:41.979214 1479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "90a530c4d9bb4117ac3ab6b1529cb250"
format_stamp: "Formatted at 2026-09-26 14:44:41 on dist-test-slave-s7k6"
server_key: "df6822b70acf56c5c98d0aa32e08269d"
server_key_iv: "1aa5f9a422c54112ba835592d21840b4"
server_key_version: "encryptionkey@0"
I20260926 14:44:41.983137 1479 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:44:41.985874 1494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:41.987313 1479 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:44:41.987437 1479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "90a530c4d9bb4117ac3ab6b1529cb250"
format_stamp: "Formatted at 2026-09-26 14:44:41 on dist-test-slave-s7k6"
server_key: "df6822b70acf56c5c98d0aa32e08269d"
server_key_iv: "1aa5f9a422c54112ba835592d21840b4"
server_key_version: "encryptionkey@0"
I20260926 14:44:41.987555 1479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:41.998036 1479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:42.001165 1479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:42.001346 1479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:42.008890 1479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44975
I20260926 14:44:42.008915 1556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44975 every 8 connection(s)
I20260926 14:44:42.009824 1479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:44:42.012717 1557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:42.012842 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 1479
I20260926 14:44:42.012944 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:44:42.013190 392 external_mini_cluster.cc:1428] Setting key f542089d20e57cefe3a7208904220cb7
I20260926 14:44:42.019788 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250: Bootstrap starting.
I20260926 14:44:42.022580 1557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250: Neither blocks nor log segments found. Creating new log.
Sep 26 14:44:42 dist-test-slave-s7k6 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:42.023494 1557 log.cc:824] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:42.026022 1557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250: No bootstrap required, opened a new log
I20260926 14:44:42.029438 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.014470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39042 (local address 127.0.98.62:44975)
0926 14:44:42.014913 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.014927 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.014964 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:44:42.015919 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.015940 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.016308 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.016621 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:42.017663 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.018702 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.019864 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.020415 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.020547 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.023277 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:42.023284 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:42.023303 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:42.023339 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:42.026044 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.026566 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.026572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.026577 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.026739 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.027017 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.027021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.027022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.027576 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:42.027769 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.028041 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.028386 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":116,"threads_started":1}
I20260926 14:44:42.030150 1557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a530c4d9bb4117ac3ab6b1529cb250" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44975 } }
I20260926 14:44:42.030432 1557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:42.030524 1557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90a530c4d9bb4117ac3ab6b1529cb250, State: Initialized, Role: FOLLOWER
I20260926 14:44:42.031015 1557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a530c4d9bb4117ac3ab6b1529cb250" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44975 } }
I20260926 14:44:42.031147 1557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:44:42.031199 1557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:44:42.031296 1557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:42.032344 1557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a530c4d9bb4117ac3ab6b1529cb250" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44975 } }
I20260926 14:44:42.032685 1557 leader_election.cc:302] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90a530c4d9bb4117ac3ab6b1529cb250; no voters:
I20260926 14:44:42.032987 1557 leader_election.cc:288] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:44:42.033147 1562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:42.033632 1562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [term 1 LEADER]: Becoming Leader. State: Replica: 90a530c4d9bb4117ac3ab6b1529cb250, State: Running, Role: LEADER
I20260926 14:44:42.033982 1562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a530c4d9bb4117ac3ab6b1529cb250" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44975 } }
I20260926 14:44:42.034210 1557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:44:42.035547 1564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90a530c4d9bb4117ac3ab6b1529cb250. Latest consensus state: current_term: 1 leader_uuid: "90a530c4d9bb4117ac3ab6b1529cb250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a530c4d9bb4117ac3ab6b1529cb250" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44975 } } }
I20260926 14:44:42.035666 1564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:42.035988 1563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90a530c4d9bb4117ac3ab6b1529cb250" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90a530c4d9bb4117ac3ab6b1529cb250" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44975 } } }
I20260926 14:44:42.036077 1563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:42.036057 1575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:44:42.039184 1575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:44:42.044740 1575 catalog_manager.cc:1347] Generated new cluster ID: 19eb005ec5e645e3b886a79d68029730
I20260926 14:44:42.044806 1575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:44:42.055408 1575 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:44:42.056350 1575 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:44:42.063469 1575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250: Generated new TSK 0
I20260926 14:44:42.064191 1575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:44:42.097684 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44975
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:42.231904 1583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:42.232180 1583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:42.232236 1583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:42.236515 1583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:42.236575 1583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:42.236615 1583 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:44:42.236668 1583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:44:42.242020 1583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:42.243103 1583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:42.244220 1583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:42.253815 1591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:42.253813 1588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:42.253945 1583 server_base.cc:1034] running on GCE node
W20260926 14:44:42.254097 1589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:42.254532 1583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:42.255174 1583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:42.256345 1583 hybrid_clock.cc:648] HybridClock initialized: now 1790433882256324 us; error 33 us; skew 500 ppm
Sep 26 14:44:42 dist-test-slave-s7k6 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:42.259047 1583 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:44:42.259886 1583 webserver.cc:460] Webserver started at http://127.0.98.1:35771/ using document root <none> and password file <none>
I20260926 14:44:42.260408 1583 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:42.260469 1583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:42.260669 1583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:42.262676 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "86b74a5f092b45a0b433455848712d07"
format_stamp: "Formatted at 2026-09-26 14:44:42 on dist-test-slave-s7k6"
server_key: "7bdebd1420545da7409c6e73da1ee763"
server_key_iv: "645d9bcdbfab21bfbb2e149a97b45469"
server_key_version: "encryptionkey@0"
I20260926 14:44:42.263296 1583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "86b74a5f092b45a0b433455848712d07"
format_stamp: "Formatted at 2026-09-26 14:44:42 on dist-test-slave-s7k6"
server_key: "7bdebd1420545da7409c6e73da1ee763"
server_key_iv: "645d9bcdbfab21bfbb2e149a97b45469"
server_key_version: "encryptionkey@0"
I20260926 14:44:42.268688 1583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260926 14:44:42.271231 1598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:42.272447 1583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:42.272562 1583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "86b74a5f092b45a0b433455848712d07"
format_stamp: "Formatted at 2026-09-26 14:44:42 on dist-test-slave-s7k6"
server_key: "7bdebd1420545da7409c6e73da1ee763"
server_key_iv: "645d9bcdbfab21bfbb2e149a97b45469"
server_key_version: "encryptionkey@0"
I20260926 14:44:42.272665 1583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:42.283221 1583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:42.286082 1583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:42.286266 1583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:42.287719 1583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:42.287765 1583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:42.287817 1583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:42.287842 1583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:44:42 dist-test-slave-s7k6 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:42.297413 1583 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41175
I20260926 14:44:42.297479 1713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41175 every 8 connection(s)
I20260926 14:44:42.298369 1583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:44:42.300556 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.288393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42523 (local address 127.0.98.62:44975)
0926 14:44:42.288530 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.288534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.289405 (+ 871us) server_negotiation.cc:407] Connection header received
0926 14:44:42.290395 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.290399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.290459 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.290519 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:42.292153 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.292833 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.294415 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.294796 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.294916 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.297253 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:42.297257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:42.297260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:42.297288 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:42.298620 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.299179 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.299182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.299184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.299261 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.299559 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.299562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.299563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.299741 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:42.299824 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.300277 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.300396 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:44:42.301276 1607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.288808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44975 (local address 127.0.98.1:42523)
0926 14:44:42.289243 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:42.289291 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:42.290152 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:42.290985 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:42.290995 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:42.291367 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:42.291992 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.292012 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.292959 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.292962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.294282 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.294293 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.294952 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.294955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.295095 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.295640 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:42.295660 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:42.297061 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:42.298719 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.298725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.298739 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.299058 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.299343 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.299346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.299348 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.299476 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.299834 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:42.299840 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:42.300065 (+ 225us) client_negotiation.cc:769] Sending connection context
0926 14:44:42.300222 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 14:44:42.300510 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
I20260926 14:44:42.302402 1714 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44975
I20260926 14:44:42.302541 1714 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:42.303059 1714 heartbeater.cc:511] Master 127.0.98.62:44975 requested a full tablet report, sending...
I20260926 14:44:42.304535 1510 ts_manager.cc:194] Registered new tserver with Master: 86b74a5f092b45a0b433455848712d07 (127.0.98.1:41175)
I20260926 14:44:42.304746 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 1583
I20260926 14:44:42.304847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:44:42.305124 392 external_mini_cluster.cc:1428] Setting key 51f4973e0a7e778d6ab64459f034cd49
I20260926 14:44:42.305950 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42523
I20260926 14:44:42.311988 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.304478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57935 (local address 127.0.98.62:44975)
0926 14:44:42.304593 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.304596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.304723 (+ 127us) server_negotiation.cc:407] Connection header received
0926 14:44:42.304824 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.304827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.304873 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.304952 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:42.305577 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.306251 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.307282 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.307574 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.307683 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.308999 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:42.309003 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:42.309006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:42.309032 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:42.310355 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.310858 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.310860 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.310861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.310938 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.311225 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.311227 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.311228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.311399 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:42.311471 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.311765 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.311865 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:44:42.312089 1720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.304364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44975 (local address 127.0.98.1:57935)
0926 14:44:42.304638 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:42.304653 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:42.304748 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:42.304980 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:42.304982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:42.305186 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:42.305319 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.305326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.306362 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.306365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.307152 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.307162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.307702 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.307706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.307796 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.308479 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:42.308494 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:42.308860 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:42.310476 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.310480 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.310482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.310742 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.311026 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.311030 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.311032 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.311132 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.311484 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:42.311487 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:42.311594 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 14:44:42.311726 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 14:44:42.311881 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:44:42.329813 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44975
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:42.437835 1723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:42.438103 1723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:42.438164 1723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:42.442174 1723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:42.442247 1723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:42.442302 1723 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:44:42.442358 1723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:44:42.447578 1723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:42.448642 1723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:42.449653 1723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:44:42.459110 1723 server_base.cc:1034] running on GCE node
W20260926 14:44:42.459095 1731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:42.459100 1729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:42.459303 1728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:42.459700 1723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:42.460278 1723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:42.461422 1723 hybrid_clock.cc:648] HybridClock initialized: now 1790433882461403 us; error 32 us; skew 500 ppm
Sep 26 14:44:42 dist-test-slave-s7k6 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:42.464068 1723 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:44:42.464956 1723 webserver.cc:460] Webserver started at http://127.0.98.2:38445/ using document root <none> and password file <none>
I20260926 14:44:42.465476 1723 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:42.465533 1723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:42.465713 1723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:42.467780 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "d5fce677d66f452a865be59b7512a699"
format_stamp: "Formatted at 2026-09-26 14:44:42 on dist-test-slave-s7k6"
server_key: "1d893f369b35eb3507a97429d58badd2"
server_key_iv: "23439002f9429abc8ed149de6c187792"
server_key_version: "encryptionkey@0"
I20260926 14:44:42.468293 1723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d5fce677d66f452a865be59b7512a699"
format_stamp: "Formatted at 2026-09-26 14:44:42 on dist-test-slave-s7k6"
server_key: "1d893f369b35eb3507a97429d58badd2"
server_key_iv: "23439002f9429abc8ed149de6c187792"
server_key_version: "encryptionkey@0"
I20260926 14:44:42.471957 1723 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:44:42.474373 1738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:42.475746 1723 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:42.475845 1723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "d5fce677d66f452a865be59b7512a699"
format_stamp: "Formatted at 2026-09-26 14:44:42 on dist-test-slave-s7k6"
server_key: "1d893f369b35eb3507a97429d58badd2"
server_key_iv: "23439002f9429abc8ed149de6c187792"
server_key_version: "encryptionkey@0"
I20260926 14:44:42.475942 1723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:42.492058 1723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:42.494920 1723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:42.495097 1723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:42.496450 1723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:42.496495 1723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:42.496532 1723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:42.496546 1723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:44:42 dist-test-slave-s7k6 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:42.506170 1723 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45853
I20260926 14:44:42.506230 1853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45853 every 8 connection(s)
I20260926 14:44:42.507174 1723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:44:42.509896 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.497255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42369 (local address 127.0.98.62:44975)
0926 14:44:42.497361 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.497365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.498399 (+ 1034us) server_negotiation.cc:407] Connection header received
0926 14:44:42.499540 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.499544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.499603 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.499705 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:42.501060 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.501682 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.503334 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.503719 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.503831 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.506389 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:42.506392 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:42.506394 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:42.506421 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:42.508032 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.508490 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.508494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.508496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.508585 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.508876 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.508880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.508882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.509069 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:42.509151 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.509613 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.509733 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:44:42.510627 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.497792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44975 (local address 127.0.98.2:42369)
0926 14:44:42.498254 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:42.498295 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:42.499318 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:42.499873 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:42.499881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:42.500288 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:42.500922 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.500941 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.501792 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.501796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.503190 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.503201 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.503860 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.503863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.504001 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.504638 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:42.504657 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:42.506219 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:42.508139 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.508145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.508158 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.508396 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.508665 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.508668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.508669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.508791 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.509160 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:42.509167 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:42.509395 (+ 228us) client_negotiation.cc:769] Sending connection context
0926 14:44:42.509554 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 14:44:42.509858 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":130,"threads_started":1}
I20260926 14:44:42.511830 1854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44975
I20260926 14:44:42.511966 1854 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:42.512432 1854 heartbeater.cc:511] Master 127.0.98.62:44975 requested a full tablet report, sending...
I20260926 14:44:42.513403 1511 ts_manager.cc:194] Registered new tserver with Master: d5fce677d66f452a865be59b7512a699 (127.0.98.2:45853)
I20260926 14:44:42.514050 1511 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42369
I20260926 14:44:42.515493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 1723
I20260926 14:44:42.515602 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:44:42.515861 392 external_mini_cluster.cc:1428] Setting key 37a3151cb11fc11f2d835e03ffa187f8
I20260926 14:44:42.520442 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.513666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44975 (local address 127.0.98.2:47707)
0926 14:44:42.513926 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:42.513937 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:42.514018 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:42.514208 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:42.514211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:42.514379 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:42.514468 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.514473 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.515211 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.515215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.516012 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.516023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.516509 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.516512 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.516598 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.517071 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:42.517084 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:42.517466 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:42.519178 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.519181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.519182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.519348 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.519596 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.519599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.519600 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.519670 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.520059 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:42.520061 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:42.520125 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 14:44:42.520208 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 14:44:42.520308 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":110,"threads_started":1}
I20260926 14:44:42.520495 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.513721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47707 (local address 127.0.98.62:44975)
0926 14:44:42.513820 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.513824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.513996 (+ 172us) server_negotiation.cc:407] Connection header received
0926 14:44:42.514089 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.514092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.514132 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.514196 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:42.514563 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.515103 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.516140 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.516406 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.516495 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.517654 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:42.517657 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:42.517659 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:42.517679 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:42.519063 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.519430 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.519433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.519435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.519520 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.519743 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.519746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.519747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.519984 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:42.520076 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.520259 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.520360 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:44:42.541872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44975
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:42.652757 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:42.653026 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:42.653071 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:42.657169 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:42.657232 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:42.657271 1863 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:44:42.657311 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:44:42.662729 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:44975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:42.663901 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:42.664973 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:42.675554 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:42.675572 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:42.675595 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:42.675815 1863 server_base.cc:1034] running on GCE node
I20260926 14:44:42.676309 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:42.676997 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:42.678162 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790433882678138 us; error 40 us; skew 500 ppm
Sep 26 14:44:42 dist-test-slave-s7k6 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:42.681010 1863 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:44:42.681849 1863 webserver.cc:460] Webserver started at http://127.0.98.3:44991/ using document root <none> and password file <none>
I20260926 14:44:42.682359 1863 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:42.682416 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:42.682577 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:42.684608 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "f6c56d22c01a423d8a201fbfd88391a2"
format_stamp: "Formatted at 2026-09-26 14:44:42 on dist-test-slave-s7k6"
server_key: "b9d438d2855522f2063ad24b824b01f3"
server_key_iv: "820afdb1bc263e3beea6fe73b64099bc"
server_key_version: "encryptionkey@0"
I20260926 14:44:42.685122 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f6c56d22c01a423d8a201fbfd88391a2"
format_stamp: "Formatted at 2026-09-26 14:44:42 on dist-test-slave-s7k6"
server_key: "b9d438d2855522f2063ad24b824b01f3"
server_key_iv: "820afdb1bc263e3beea6fe73b64099bc"
server_key_version: "encryptionkey@0"
I20260926 14:44:42.688838 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:44:42.691213 1878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:42.692456 1863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:42.692546 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "f6c56d22c01a423d8a201fbfd88391a2"
format_stamp: "Formatted at 2026-09-26 14:44:42 on dist-test-slave-s7k6"
server_key: "b9d438d2855522f2063ad24b824b01f3"
server_key_iv: "820afdb1bc263e3beea6fe73b64099bc"
server_key_version: "encryptionkey@0"
I20260926 14:44:42.692637 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:42.722929 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:42.726142 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:42.726315 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:42.727794 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:42.727845 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:42.727885 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:42.727912 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:44:42 dist-test-slave-s7k6 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:42.738095 1863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33145
I20260926 14:44:42.738270 1993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33145 every 8 connection(s)
I20260926 14:44:42.739028 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:44:42.741055 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.728713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57617 (local address 127.0.98.62:44975)
0926 14:44:42.728831 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.728834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.729967 (+ 1133us) server_negotiation.cc:407] Connection header received
0926 14:44:42.731118 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.731121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.731177 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.731250 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:42.732710 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.733515 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.734683 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.734977 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.735080 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.737646 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:42.737649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:42.737651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:42.737675 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:42.739198 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.739782 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.739786 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.739788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.739871 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.740156 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.740158 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.740159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.740323 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:42.740394 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.740835 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.740936 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:44:42.741818 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.729291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44975 (local address 127.0.98.3:57617)
0926 14:44:42.729805 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:42.729847 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:42.730884 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:42.731445 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:42.731452 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:42.731878 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:42.732546 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.732566 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.733653 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.733656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.734578 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.734590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.735117 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.735122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.735313 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.735827 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:42.735844 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:42.737478 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:42.739305 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.739310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.739325 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.739658 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.739938 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.739940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.739942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.740083 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.740404 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:42.740409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:42.740641 (+ 232us) client_negotiation.cc:769] Sending connection context
0926 14:44:42.740794 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 14:44:42.741033 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":146,"threads_started":1}
I20260926 14:44:42.743017 1994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44975
I20260926 14:44:42.743155 1994 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:42.743659 1994 heartbeater.cc:511] Master 127.0.98.62:44975 requested a full tablet report, sending...
I20260926 14:44:42.744724 1510 ts_manager.cc:194] Registered new tserver with Master: f6c56d22c01a423d8a201fbfd88391a2 (127.0.98.3:33145)
I20260926 14:44:42.745463 1510 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57617
I20260926 14:44:42.747886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 1863
I20260926 14:44:42.748010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:44:42.748332 392 external_mini_cluster.cc:1428] Setting key 93fe12f8af7f08d82c10f861a8612bd9
I20260926 14:44:42.750353 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:44:42.752170 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.744999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51475 (local address 127.0.98.62:44975)
0926 14:44:42.745126 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.745130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.745299 (+ 169us) server_negotiation.cc:407] Connection header received
0926 14:44:42.745405 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.745407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.745443 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.745507 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:42.745950 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.746488 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.747563 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.747807 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.747885 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.749046 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:42.749048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:42.749051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:42.749074 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:42.750683 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.751093 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.751097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.751099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.751179 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.751416 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.751420 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.751421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.751595 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:42.751684 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.751916 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.752019 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:44:42.752187 2000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.744950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44975 (local address 127.0.98.3:51475)
0926 14:44:42.745218 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:42.745233 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:42.745330 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:42.745544 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:42.745547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:42.745753 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:42.745851 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.745857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.746619 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.746623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.747455 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.747465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.747917 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.747920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.748022 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.748527 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:42.748541 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:42.748915 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:42.750806 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.750809 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.750811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.751006 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.751256 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:42.751260 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:42.751262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:42.751338 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:42.751684 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:42.751687 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:42.751770 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 14:44:42.751853 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 14:44:42.751996 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
Sep 26 14:44:42 dist-test-slave-s7k6 krb5kdc[1466](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:44:42 dist-test-slave-s7k6 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:42.774387 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.766815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.0.98.62:44975)
0926 14:44:42.766932 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.766936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.767194 (+ 258us) server_negotiation.cc:407] Connection header received
0926 14:44:42.767322 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.767324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.767362 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.767434 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:42.768038 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.768553 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.769336 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.769531 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.769596 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.771883 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:42.771886 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:42.771888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:42.771910 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:42.773162 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.773509 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.773511 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.773512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.773575 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:42.773756 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:42.773757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:42.773758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:42.773917 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:42.774007 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.774183 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.774269 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:44:42.777051 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:44:42.780130 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:44:42.794876 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.789357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33145 (local address 127.0.0.1:34504)
0926 14:44:42.790085 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:42.790096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:42.790181 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:42.791003 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:42.791006 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:42.791023 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:42.791119 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.791124 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.792454 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.792457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.793549 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.793556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.794298 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.794302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.794415 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.794454 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 14:44:42.794563 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 14:44:42.794688 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":101,"threads_started":1}
I20260926 14:44:42.794875 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.789754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.0.98.3:33145)
0926 14:44:42.790396 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.790407 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.790427 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:44:42.790495 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.790501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.790708 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.790873 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:42.791261 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.792324 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.793643 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.794185 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.794284 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.794457 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.794601 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.794706 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":84,"threads_started":1}
I20260926 14:44:42.795212 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.788894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53268 (local address 127.0.98.1:41175)
0926 14:44:42.789279 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.789294 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.789666 (+ 372us) server_negotiation.cc:407] Connection header received
0926 14:44:42.789892 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.789897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.790113 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.790257 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:42.790694 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.791736 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.793455 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.794255 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.794364 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.794527 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.794984 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.795083 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":80,"threads_started":1}
I20260926 14:44:42.795131 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.788608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45853 (local address 127.0.0.1:42290)
0926 14:44:42.789123 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:42.789176 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:42.789303 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:42.790277 (+ 974us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:42.790283 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:42.790303 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:42.790455 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.790462 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.791865 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.791868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.793456 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.793467 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.794589 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.794593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.794774 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.794850 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 14:44:42.794920 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 14:44:42.794991 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"mutex_wait_us":124,"thread_start_us":91,"threads_started":1}
I20260926 14:44:42.795478 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.789331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.0.98.2:45853)
0926 14:44:42.789786 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.789798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.789818 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:44:42.789891 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.789897 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.790113 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.790257 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:42.790577 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.791736 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.793583 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.794326 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.794408 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.794533 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.795243 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.795345 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":118,"threads_started":1}
I20260926 14:44:42.795574 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.788739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41175 (local address 127.0.0.1:53268)
0926 14:44:42.789561 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:42.789575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:42.789747 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:42.790424 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:42.790426 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:42.790464 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:42.790575 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.790581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.791837 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.791840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.793336 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.793346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.794589 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.794593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.794804 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.794853 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 14:44:42.795370 (+ 517us) client_negotiation.cc:240] Negotiation successful
0926 14:44:42.795461 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"mutex_wait_us":145,"thread_start_us":116,"threads_started":1}
I20260926 14:44:42.798241 1648 tablet_service.cc:1467] Processing CreateTablet for tablet d3bbb4382ec5432394865f220c08ba61 (DEFAULT_TABLE table=test-table [id=ce697b228ef3452bbe79d15f21f9c3f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:42.798267 1928 tablet_service.cc:1467] Processing CreateTablet for tablet d3bbb4382ec5432394865f220c08ba61 (DEFAULT_TABLE table=test-table [id=ce697b228ef3452bbe79d15f21f9c3f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:42.798267 1788 tablet_service.cc:1467] Processing CreateTablet for tablet d3bbb4382ec5432394865f220c08ba61 (DEFAULT_TABLE table=test-table [id=ce697b228ef3452bbe79d15f21f9c3f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:42.799250 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3bbb4382ec5432394865f220c08ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:42.799273 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3bbb4382ec5432394865f220c08ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:42.799283 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3bbb4382ec5432394865f220c08ba61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:42.806164 2017 tablet_bootstrap.cc:492] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07: Bootstrap starting.
I20260926 14:44:42.806913 2018 tablet_bootstrap.cc:492] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699: Bootstrap starting.
I20260926 14:44:42.807749 2019 tablet_bootstrap.cc:492] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: Bootstrap starting.
I20260926 14:44:42.809197 2017 tablet_bootstrap.cc:654] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:42.809197 2018 tablet_bootstrap.cc:654] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:42.809464 2019 tablet_bootstrap.cc:654] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:42.810086 2018 log.cc:824] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:42.810086 2017 log.cc:824] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:42.810114 2019 log.cc:824] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:42.812191 2018 tablet_bootstrap.cc:492] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699: No bootstrap required, opened a new log
I20260926 14:44:42.812191 2017 tablet_bootstrap.cc:492] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07: No bootstrap required, opened a new log
I20260926 14:44:42.812397 2017 ts_tablet_manager.cc:1397] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:44:42.812397 2018 ts_tablet_manager.cc:1397] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:44:42.812548 2019 tablet_bootstrap.cc:492] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: No bootstrap required, opened a new log
I20260926 14:44:42.812726 2019 ts_tablet_manager.cc:1397] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:44:42.815647 2019 raft_consensus.cc:348] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:42.815893 2019 raft_consensus.cc:374] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:42.815963 2019 raft_consensus.cc:729] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c56d22c01a423d8a201fbfd88391a2, State: Initialized, Role: FOLLOWER
I20260926 14:44:42.816252 2017 raft_consensus.cc:348] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:42.816252 2018 raft_consensus.cc:348] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:42.816421 2019 consensus_queue.cc:260] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:42.816501 2018 raft_consensus.cc:374] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:42.816501 2017 raft_consensus.cc:374] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:42.816558 2017 raft_consensus.cc:729] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b74a5f092b45a0b433455848712d07, State: Initialized, Role: FOLLOWER
I20260926 14:44:42.816588 2018 raft_consensus.cc:729] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5fce677d66f452a865be59b7512a699, State: Initialized, Role: FOLLOWER
I20260926 14:44:42.817094 2018 consensus_queue.cc:260] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:42.817094 2017 consensus_queue.cc:260] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:42.817359 2019 ts_tablet_manager.cc:1428] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:44:42.817713 1994 heartbeater.cc:503] Master 127.0.98.62:44975 was elected leader, sending a full tablet report...
I20260926 14:44:42.817713 1714 heartbeater.cc:503] Master 127.0.98.62:44975 was elected leader, sending a full tablet report...
I20260926 14:44:42.819136 2018 ts_tablet_manager.cc:1428] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 14:44:42.819157 2017 ts_tablet_manager.cc:1428] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 14:44:42.819530 1854 heartbeater.cc:503] Master 127.0.98.62:44975 was elected leader, sending a full tablet report...
I20260926 14:44:42.936918 2023 raft_consensus.cc:482] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:44:42.937146 2023 raft_consensus.cc:504] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:42.938400 2023 leader_election.cc:288] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5fce677d66f452a865be59b7512a699 (127.0.98.2:45853), 86b74a5f092b45a0b433455848712d07 (127.0.98.1:41175)
I20260926 14:44:42.942096 1887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.938708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45853 (local address 127.0.98.3:58625)
0926 14:44:42.938907 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:42.938922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:42.939024 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:42.939296 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:42.939299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:42.939317 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:42.939421 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.939426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.940178 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.940182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.941000 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.941011 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.941675 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.941679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.941784 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.941802 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:44:42.941870 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:44:42.941950 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260926 14:44:42.942096 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.938812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58625 (local address 127.0.98.2:45853)
0926 14:44:42.938922 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.938927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.939086 (+ 159us) server_negotiation.cc:407] Connection header received
0926 14:44:42.939144 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.939146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.939200 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.939265 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:42.939534 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.940064 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.941147 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.941556 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.941652 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.941717 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.941903 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.941979 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:44:42.942739 2026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.938743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41175 (local address 127.0.98.3:45153)
0926 14:44:42.939269 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:42.939282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:42.939381 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:42.939611 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:42.939613 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:42.939628 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:42.939748 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.939759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.940443 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.940446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.941599 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:42.941610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.942366 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.942370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:42.942447 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.942466 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:44:42.942533 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:44:42.942601 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":136,"threads_started":1}
I20260926 14:44:42.942796 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.939117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45153 (local address 127.0.98.1:41175)
0926 14:44:42.939235 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.939239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.939353 (+ 114us) server_negotiation.cc:407] Connection header received
0926 14:44:42.939478 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.939480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.939525 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.939589 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:42.939851 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.940333 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.941730 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.942126 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.942222 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.942290 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.942579 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.942662 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:44:42.942833 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3bbb4382ec5432394865f220c08ba61" candidate_uuid: "f6c56d22c01a423d8a201fbfd88391a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5fce677d66f452a865be59b7512a699" is_pre_election: true
I20260926 14:44:42.943228 1808 raft_consensus.cc:2454] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c56d22c01a423d8a201fbfd88391a2 in term 0.
I20260926 14:44:42.943571 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3bbb4382ec5432394865f220c08ba61" candidate_uuid: "f6c56d22c01a423d8a201fbfd88391a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b74a5f092b45a0b433455848712d07" is_pre_election: true
I20260926 14:44:42.943769 1881 leader_election.cc:302] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5fce677d66f452a865be59b7512a699, f6c56d22c01a423d8a201fbfd88391a2; no voters:
I20260926 14:44:42.943915 1668 raft_consensus.cc:2454] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6c56d22c01a423d8a201fbfd88391a2 in term 0.
I20260926 14:44:42.944079 2023 raft_consensus.cc:2781] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:44:42.944200 2023 raft_consensus.cc:482] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:44:42.944281 2023 raft_consensus.cc:3037] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:42.945361 2023 raft_consensus.cc:504] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:42.945724 2023 leader_election.cc:288] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [CANDIDATE]: Term 1 election: Requested vote from peers d5fce677d66f452a865be59b7512a699 (127.0.98.2:45853), 86b74a5f092b45a0b433455848712d07 (127.0.98.1:41175)
I20260926 14:44:42.946121 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3bbb4382ec5432394865f220c08ba61" candidate_uuid: "f6c56d22c01a423d8a201fbfd88391a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5fce677d66f452a865be59b7512a699"
I20260926 14:44:42.946251 1808 raft_consensus.cc:3037] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:42.946233 1668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3bbb4382ec5432394865f220c08ba61" candidate_uuid: "f6c56d22c01a423d8a201fbfd88391a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86b74a5f092b45a0b433455848712d07"
I20260926 14:44:42.946369 1668 raft_consensus.cc:3037] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:42.947407 1808 raft_consensus.cc:2454] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c56d22c01a423d8a201fbfd88391a2 in term 1.
I20260926 14:44:42.947419 1668 raft_consensus.cc:2454] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6c56d22c01a423d8a201fbfd88391a2 in term 1.
I20260926 14:44:42.947731 1881 leader_election.cc:302] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5fce677d66f452a865be59b7512a699, f6c56d22c01a423d8a201fbfd88391a2; no voters:
I20260926 14:44:42.947909 2023 raft_consensus.cc:2781] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:42.948129 2023 raft_consensus.cc:686] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 1 LEADER]: Becoming Leader. State: Replica: f6c56d22c01a423d8a201fbfd88391a2, State: Running, Role: LEADER
I20260926 14:44:42.948459 2023 consensus_queue.cc:237] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:42.951809 1509 catalog_manager.cc:5495] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6c56d22c01a423d8a201fbfd88391a2 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f6c56d22c01a423d8a201fbfd88391a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } health_report { overall_health: HEALTHY } } }
I20260926 14:44:42.989423 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:42.985757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.0.98.3:33145)
0926 14:44:42.985888 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:42.985892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:42.985907 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:44:42.986064 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:42.986066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:42.986123 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:42.986240 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:44:42.986552 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.987104 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.988066 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:42.988289 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:42.988370 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:42.988638 (+ 268us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:44:42.989075 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:44:42.989159 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:42.989254 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 14:44:42.989307 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 14:44:42.989959 1995 tablet.cc:2369] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:44:42.995503 1808 raft_consensus.cc:1261] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699 [term 1 FOLLOWER]: Refusing update from remote peer f6c56d22c01a423d8a201fbfd88391a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:42.995502 1668 raft_consensus.cc:1261] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07 [term 1 FOLLOWER]: Refusing update from remote peer f6c56d22c01a423d8a201fbfd88391a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:42.996222 2023 consensus_queue.cc:1035] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:42.996418 2027 consensus_queue.cc:1035] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:43.004001 2032 mvcc.cc:205] Tried to move back new op lower bound from 7333617184742387712 to 7333617184559370240. Current Snapshot: MvccSnapshot[applied={T|T < 7333617184742387712}]
I20260926 14:44:43.004526 2031 mvcc.cc:205] Tried to move back new op lower bound from 7333617184742387712 to 7333617184559370240. Current Snapshot: MvccSnapshot[applied={T|T < 7333617184742387712}]
I20260926 14:44:43.015828 1510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:43.016008 1510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:43.018572 1510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250: Sending DeleteTablet for 3 replicas of tablet d3bbb4382ec5432394865f220c08ba61
I20260926 14:44:43.019310 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet d3bbb4382ec5432394865f220c08ba61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42290
I20260926 14:44:43.019385 1648 tablet_service.cc:1514] Processing DeleteTablet for tablet d3bbb4382ec5432394865f220c08ba61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53268
I20260926 14:44:43.020777 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet d3bbb4382ec5432394865f220c08ba61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34504
I20260926 14:44:43.021274 2042 tablet_replica.cc:330] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: stopping tablet replica
I20260926 14:44:43.021535 1510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:44:43.021634 2042 raft_consensus.cc:2229] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 1 LEADER]: Raft consensus shutting down.
W20260926 14:44:43.021965 1510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:44:43.022024 2042 raft_consensus.cc:2258] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:43.022305 2040 tablet_replica.cc:330] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699: stopping tablet replica
I20260926 14:44:43.022593 2040 raft_consensus.cc:2229] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:43.022914 2040 raft_consensus.cc:2258] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:43.023524 2041 tablet_replica.cc:330] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07: stopping tablet replica
I20260926 14:44:43.023900 2041 raft_consensus.cc:2229] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:43.024046 2042 ts_tablet_manager.cc:1905] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:43.024260 2041 raft_consensus.cc:2258] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:43.025857 2040 ts_tablet_manager.cc:1905] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:43.027158 2042 ts_tablet_manager.cc:1918] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:44:43.027323 2042 log.cc:1196] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/d3bbb4382ec5432394865f220c08ba61
I20260926 14:44:43.027745 2042 ts_tablet_manager.cc:1939] T d3bbb4382ec5432394865f220c08ba61 P f6c56d22c01a423d8a201fbfd88391a2: Deleting consensus metadata
I20260926 14:44:43.027707 1648 tablet_service.cc:1467] Processing CreateTablet for tablet c733ff94643c4b78b44ed4623821a3e5 (DEFAULT_TABLE table=test-table [id=f0eb645f5d4740af8c0ef600c0bb3e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:43.027968 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c733ff94643c4b78b44ed4623821a3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:43.028527 2040 ts_tablet_manager.cc:1918] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:44:43.028614 2040 log.cc:1196] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/d3bbb4382ec5432394865f220c08ba61
I20260926 14:44:43.028877 2040 ts_tablet_manager.cc:1939] T d3bbb4382ec5432394865f220c08ba61 P d5fce677d66f452a865be59b7512a699: Deleting consensus metadata
I20260926 14:44:43.030315 2017 tablet_bootstrap.cc:492] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07: Bootstrap starting.
I20260926 14:44:43.030782 1497 catalog_manager.cc:4841] TS d5fce677d66f452a865be59b7512a699 (127.0.98.2:45853): tablet d3bbb4382ec5432394865f220c08ba61 (table test-table [id=ce697b228ef3452bbe79d15f21f9c3f9]) successfully deleted
I20260926 14:44:43.030944 1497 catalog_manager.cc:4841] TS f6c56d22c01a423d8a201fbfd88391a2 (127.0.98.3:33145): tablet d3bbb4382ec5432394865f220c08ba61 (table test-table [id=ce697b228ef3452bbe79d15f21f9c3f9]) successfully deleted
I20260926 14:44:43.031102 1788 tablet_service.cc:1467] Processing CreateTablet for tablet c733ff94643c4b78b44ed4623821a3e5 (DEFAULT_TABLE table=test-table [id=f0eb645f5d4740af8c0ef600c0bb3e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:43.031282 2017 tablet_bootstrap.cc:654] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:43.031335 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c733ff94643c4b78b44ed4623821a3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:43.031344 1928 tablet_service.cc:1467] Processing CreateTablet for tablet c733ff94643c4b78b44ed4623821a3e5 (DEFAULT_TABLE table=test-table [id=f0eb645f5d4740af8c0ef600c0bb3e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:43.031569 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c733ff94643c4b78b44ed4623821a3e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:43.033210 2018 tablet_bootstrap.cc:492] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699: Bootstrap starting.
I20260926 14:44:43.033737 2019 tablet_bootstrap.cc:492] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2: Bootstrap starting.
I20260926 14:44:43.034070 2018 tablet_bootstrap.cc:654] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:43.034071 2017 tablet_bootstrap.cc:492] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07: No bootstrap required, opened a new log
I20260926 14:44:43.034148 2017 ts_tablet_manager.cc:1397] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 14:44:43.034633 2019 tablet_bootstrap.cc:654] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:43.034807 2017 raft_consensus.cc:348] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.034924 2017 raft_consensus.cc:374] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:43.034956 2017 raft_consensus.cc:729] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b74a5f092b45a0b433455848712d07, State: Initialized, Role: FOLLOWER
I20260926 14:44:43.035079 2017 consensus_queue.cc:260] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.035378 2041 ts_tablet_manager.cc:1905] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:43.035845 2017 ts_tablet_manager.cc:1428] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:44:43.036115 2018 tablet_bootstrap.cc:492] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699: No bootstrap required, opened a new log
I20260926 14:44:43.036192 2018 ts_tablet_manager.cc:1397] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:44:43.036588 2018 raft_consensus.cc:348] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.036686 2018 raft_consensus.cc:374] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:43.036713 2018 raft_consensus.cc:729] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5fce677d66f452a865be59b7512a699, State: Initialized, Role: FOLLOWER
I20260926 14:44:43.036842 2018 consensus_queue.cc:260] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.037062 2018 ts_tablet_manager.cc:1428] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:44:43.038118 2019 tablet_bootstrap.cc:492] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2: No bootstrap required, opened a new log
I20260926 14:44:43.038211 2019 ts_tablet_manager.cc:1397] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260926 14:44:43.038637 2041 ts_tablet_manager.cc:1918] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:44:43.038727 2041 log.cc:1196] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/wal/wals/d3bbb4382ec5432394865f220c08ba61
I20260926 14:44:43.038662 2019 raft_consensus.cc:348] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.038789 2019 raft_consensus.cc:374] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:43.038825 2019 raft_consensus.cc:729] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c56d22c01a423d8a201fbfd88391a2, State: Initialized, Role: FOLLOWER
I20260926 14:44:43.038949 2019 consensus_queue.cc:260] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.039139 2041 ts_tablet_manager.cc:1939] T d3bbb4382ec5432394865f220c08ba61 P 86b74a5f092b45a0b433455848712d07: Deleting consensus metadata
I20260926 14:44:43.039222 2019 ts_tablet_manager.cc:1428] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:44:43.040330 1498 catalog_manager.cc:4841] TS 86b74a5f092b45a0b433455848712d07 (127.0.98.1:41175): tablet d3bbb4382ec5432394865f220c08ba61 (table test-table [id=ce697b228ef3452bbe79d15f21f9c3f9]) successfully deleted
W20260926 14:44:43.049724 1715 tablet.cc:2369] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:44:43.072727 2024 raft_consensus.cc:482] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:44:43.072875 2024 raft_consensus.cc:504] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.073908 2024 leader_election.cc:288] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5fce677d66f452a865be59b7512a699 (127.0.98.2:45853), f6c56d22c01a423d8a201fbfd88391a2 (127.0.98.3:33145)
I20260926 14:44:43.079190 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.074640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38685 (local address 127.0.98.3:33145)
0926 14:44:43.074821 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:43.074826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:43.074912 (+ 86us) server_negotiation.cc:407] Connection header received
0926 14:44:43.075021 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:43.075024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:43.075071 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:43.075149 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:43.075411 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.075990 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.077757 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.078410 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.078532 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.078628 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:43.078999 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 14:44:43.079069 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 14:44:43.079213 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.074512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33145 (local address 127.0.98.1:38685)
0926 14:44:43.074831 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:43.074844 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:43.074931 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:43.075165 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:43.075167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:43.075187 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:43.075281 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:43.075288 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.076118 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.076121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:43.077616 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:43.077628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.078660 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.078663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:43.078843 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.078874 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 14:44:43.078951 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 14:44:43.079037 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":74,"thread_start_us":97,"threads_started":1}
I20260926 14:44:43.079996 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c733ff94643c4b78b44ed4623821a3e5" candidate_uuid: "86b74a5f092b45a0b433455848712d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c56d22c01a423d8a201fbfd88391a2" is_pre_election: true
I20260926 14:44:43.080252 1948 raft_consensus.cc:2454] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b74a5f092b45a0b433455848712d07 in term 0.
I20260926 14:44:43.080343 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.074254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36213 (local address 127.0.98.2:45853)
0926 14:44:43.074395 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:43.074400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:43.074502 (+ 102us) server_negotiation.cc:407] Connection header received
0926 14:44:43.074804 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:43.074807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:43.074863 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:43.074912 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:43.075278 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.076103 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.078704 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.079370 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.079474 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.079533 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:43.080133 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 14:44:43.080221 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 14:44:43.080819 2046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.074146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45853 (local address 127.0.98.1:36213)
0926 14:44:43.074403 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:43.074418 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:43.074583 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:43.075048 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:43.075050 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:43.075064 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:43.075176 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:43.075182 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.076205 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.076208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:43.077616 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:43.077628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.079869 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.079873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:43.079987 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.080012 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:44:43.080091 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:44:43.080166 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":83,"thread_start_us":88,"threads_started":1}
I20260926 14:44:43.080808 1601 leader_election.cc:302] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b74a5f092b45a0b433455848712d07, f6c56d22c01a423d8a201fbfd88391a2; no voters:
I20260926 14:44:43.081117 2024 raft_consensus.cc:2781] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:44:43.081174 2024 raft_consensus.cc:482] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:44:43.081211 2024 raft_consensus.cc:3037] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:43.081401 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c733ff94643c4b78b44ed4623821a3e5" candidate_uuid: "86b74a5f092b45a0b433455848712d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5fce677d66f452a865be59b7512a699" is_pre_election: true
I20260926 14:44:43.081519 1808 raft_consensus.cc:2454] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b74a5f092b45a0b433455848712d07 in term 0.
I20260926 14:44:43.082477 2024 raft_consensus.cc:504] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.082892 2024 leader_election.cc:288] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [CANDIDATE]: Term 1 election: Requested vote from peers d5fce677d66f452a865be59b7512a699 (127.0.98.2:45853), f6c56d22c01a423d8a201fbfd88391a2 (127.0.98.3:33145)
I20260926 14:44:43.083225 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c733ff94643c4b78b44ed4623821a3e5" candidate_uuid: "86b74a5f092b45a0b433455848712d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5fce677d66f452a865be59b7512a699"
I20260926 14:44:43.083341 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c733ff94643c4b78b44ed4623821a3e5" candidate_uuid: "86b74a5f092b45a0b433455848712d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c56d22c01a423d8a201fbfd88391a2"
I20260926 14:44:43.083385 1808 raft_consensus.cc:3037] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:43.083452 1948 raft_consensus.cc:3037] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:43.084694 1808 raft_consensus.cc:2454] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b74a5f092b45a0b433455848712d07 in term 1.
I20260926 14:44:43.084694 1948 raft_consensus.cc:2454] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b74a5f092b45a0b433455848712d07 in term 1.
I20260926 14:44:43.085006 1601 leader_election.cc:302] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b74a5f092b45a0b433455848712d07, f6c56d22c01a423d8a201fbfd88391a2; no voters:
I20260926 14:44:43.085321 2024 raft_consensus.cc:2781] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:43.085546 2024 raft_consensus.cc:686] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 1 LEADER]: Becoming Leader. State: Replica: 86b74a5f092b45a0b433455848712d07, State: Running, Role: LEADER
I20260926 14:44:43.085822 2024 consensus_queue.cc:237] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.088666 1509 catalog_manager.cc:5495] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86b74a5f092b45a0b433455848712d07 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "86b74a5f092b45a0b433455848712d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } health_report { overall_health: UNKNOWN } } }
Sep 26 14:44:43 dist-test-slave-s7k6 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:43.141688 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.132653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39064 (local address 127.0.98.62:44975)
0926 14:44:43.132783 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:43.132786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:43.133593 (+ 807us) server_negotiation.cc:407] Connection header received
0926 14:44:43.133725 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:43.133728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:43.133768 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:43.133849 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:43.134556 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.135175 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.136046 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.136260 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.136345 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.138806 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:43.138810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:43.138813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:43.138836 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:43.140122 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:43.140553 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:43.140557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:43.140559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:43.140626 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:43.140858 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:43.140861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:43.140863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:43.141030 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:43.141110 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:43.141387 (+ 277us) server_negotiation.cc:299] Negotiation successful
0926 14:44:43.141547 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":58}
I20260926 14:44:43.141697 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.132566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44975 (local address 127.0.0.1:39064)
0926 14:44:43.133498 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:43.133512 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:43.133637 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:43.133882 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:43.133885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:43.134312 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:43.134439 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:43.134448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.135283 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.135286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:43.135933 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:43.135944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.136356 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.136359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:43.136431 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.136999 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:43.137010 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:43.138619 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:43.140234 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:43.140244 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:43.140251 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:43.140468 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:43.140709 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:43.140712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:43.140713 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:43.140786 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:43.141122 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:43.141129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:43.141239 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 14:44:43.141333 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 14:44:43.141490 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":120,"threads_started":1}
I20260926 14:44:43.146279 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39064:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:44:43.147141 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:44:43.152415 1648 tablet_service.cc:1467] Processing CreateTablet for tablet d7b8379a0fe04390a3ba70e0489bb03a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bdd1625fc3e4ae5b78188b52905d216]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:44:43.152661 1928 tablet_service.cc:1467] Processing CreateTablet for tablet d7b8379a0fe04390a3ba70e0489bb03a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bdd1625fc3e4ae5b78188b52905d216]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:44:43.152673 1788 tablet_service.cc:1467] Processing CreateTablet for tablet d7b8379a0fe04390a3ba70e0489bb03a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bdd1625fc3e4ae5b78188b52905d216]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:44:43.152854 1648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b8379a0fe04390a3ba70e0489bb03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:43.153005 1928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b8379a0fe04390a3ba70e0489bb03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:43.153059 1788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b8379a0fe04390a3ba70e0489bb03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:43.155675 2019 tablet_bootstrap.cc:492] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2: Bootstrap starting.
I20260926 14:44:43.155762 2017 tablet_bootstrap.cc:492] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: Bootstrap starting.
I20260926 14:44:43.156533 2019 tablet_bootstrap.cc:654] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:43.156652 2017 tablet_bootstrap.cc:654] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:43.157590 2018 tablet_bootstrap.cc:492] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699: Bootstrap starting.
I20260926 14:44:43.157720 2019 tablet_bootstrap.cc:492] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2: No bootstrap required, opened a new log
I20260926 14:44:43.157794 2019 ts_tablet_manager.cc:1397] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:44:43.158241 2019 raft_consensus.cc:348] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.158354 2019 raft_consensus.cc:374] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:43.158385 2019 raft_consensus.cc:729] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6c56d22c01a423d8a201fbfd88391a2, State: Initialized, Role: FOLLOWER
I20260926 14:44:43.158576 2018 tablet_bootstrap.cc:654] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:43.158514 2019 consensus_queue.cc:260] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.158824 2019 ts_tablet_manager.cc:1428] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:44:43.158916 2035 tablet_replica.cc:439] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } } }
I20260926 14:44:43.159049 2035 tablet_replica.cc:442] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:43.159950 2017 tablet_bootstrap.cc:492] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: No bootstrap required, opened a new log
I20260926 14:44:43.160043 2017 ts_tablet_manager.cc:1397] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 14:44:43.160059 2018 tablet_bootstrap.cc:492] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699: No bootstrap required, opened a new log
I20260926 14:44:43.160131 2018 ts_tablet_manager.cc:1397] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 14:44:43.160454 2018 raft_consensus.cc:348] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.160537 2018 raft_consensus.cc:374] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:43.160465 2017 raft_consensus.cc:348] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.160557 2018 raft_consensus.cc:729] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5fce677d66f452a865be59b7512a699, State: Initialized, Role: FOLLOWER
I20260926 14:44:43.160573 2017 raft_consensus.cc:374] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:43.160599 2017 raft_consensus.cc:729] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b74a5f092b45a0b433455848712d07, State: Initialized, Role: FOLLOWER
I20260926 14:44:43.160658 2018 consensus_queue.cc:260] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.160706 2017 consensus_queue.cc:260] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.160897 2018 ts_tablet_manager.cc:1428] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:44:43.160933 2017 ts_tablet_manager.cc:1428] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:44:43.160881 2025 tablet_replica.cc:439] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } } }
I20260926 14:44:43.161018 2025 tablet_replica.cc:442] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:43.161129 2024 tablet_replica.cc:439] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } } }
I20260926 14:44:43.161247 2024 tablet_replica.cc:442] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 14:44:43.258930 1855 tablet.cc:2369] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:44:43.260587 2024 raft_consensus.cc:482] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:44:43.260720 2024 raft_consensus.cc:504] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.261140 2024 leader_election.cc:288] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d5fce677d66f452a865be59b7512a699 (127.0.98.2:45853), f6c56d22c01a423d8a201fbfd88391a2 (127.0.98.3:33145)
I20260926 14:44:43.261600 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b8379a0fe04390a3ba70e0489bb03a" candidate_uuid: "86b74a5f092b45a0b433455848712d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5fce677d66f452a865be59b7512a699" is_pre_election: true
I20260926 14:44:43.261623 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b8379a0fe04390a3ba70e0489bb03a" candidate_uuid: "86b74a5f092b45a0b433455848712d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c56d22c01a423d8a201fbfd88391a2" is_pre_election: true
I20260926 14:44:43.261761 1808 raft_consensus.cc:2454] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b74a5f092b45a0b433455848712d07 in term 0.
I20260926 14:44:43.261761 1948 raft_consensus.cc:2454] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 86b74a5f092b45a0b433455848712d07 in term 0.
I20260926 14:44:43.262032 1601 leader_election.cc:302] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b74a5f092b45a0b433455848712d07, d5fce677d66f452a865be59b7512a699; no voters:
I20260926 14:44:43.262205 2024 raft_consensus.cc:2781] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:44:43.262269 2024 raft_consensus.cc:482] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:44:43.262315 2024 raft_consensus.cc:3037] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:43.263301 2024 raft_consensus.cc:504] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.263670 2024 leader_election.cc:288] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [CANDIDATE]: Term 1 election: Requested vote from peers d5fce677d66f452a865be59b7512a699 (127.0.98.2:45853), f6c56d22c01a423d8a201fbfd88391a2 (127.0.98.3:33145)
I20260926 14:44:43.263988 1808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b8379a0fe04390a3ba70e0489bb03a" candidate_uuid: "86b74a5f092b45a0b433455848712d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5fce677d66f452a865be59b7512a699"
I20260926 14:44:43.264142 1808 raft_consensus.cc:3037] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:43.264145 1948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b8379a0fe04390a3ba70e0489bb03a" candidate_uuid: "86b74a5f092b45a0b433455848712d07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6c56d22c01a423d8a201fbfd88391a2"
I20260926 14:44:43.264293 1948 raft_consensus.cc:3037] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:43.265322 1808 raft_consensus.cc:2454] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b74a5f092b45a0b433455848712d07 in term 1.
I20260926 14:44:43.265393 1948 raft_consensus.cc:2454] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 86b74a5f092b45a0b433455848712d07 in term 1.
I20260926 14:44:43.265618 1601 leader_election.cc:302] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86b74a5f092b45a0b433455848712d07, d5fce677d66f452a865be59b7512a699; no voters:
I20260926 14:44:43.265820 2024 raft_consensus.cc:2781] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:43.265945 2024 raft_consensus.cc:686] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [term 1 LEADER]: Becoming Leader. State: Replica: 86b74a5f092b45a0b433455848712d07, State: Running, Role: LEADER
I20260926 14:44:43.266083 2024 consensus_queue.cc:237] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } }
I20260926 14:44:43.266789 2048 tablet_replica.cc:439] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: TxnStatusTablet state changed. Reason: New leader 86b74a5f092b45a0b433455848712d07. Latest consensus state: current_term: 1 leader_uuid: "86b74a5f092b45a0b433455848712d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } } }
I20260926 14:44:43.266906 2048 tablet_replica.cc:442] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:44:43.267179 2060 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:44:43.267340 2060 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:44:43.267935 1508 catalog_manager.cc:5495] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 86b74a5f092b45a0b433455848712d07 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "86b74a5f092b45a0b433455848712d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } health_report { overall_health: UNKNOWN } } }
I20260926 14:44:43.292488 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.288638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41175 (local address 127.0.0.1:53274)
0926 14:44:43.288788 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:43.288805 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:43.288916 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:43.289233 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:43.289236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:43.289245 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:44:43.289351 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:43.289357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.290074 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.290078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:43.290891 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:43.290902 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.291329 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.291333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:43.291443 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.291489 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:44:43.292186 (+ 697us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:44:43.292209 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:44:43.292267 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:44:43.292356 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 14:44:43.292510 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.288755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53274 (local address 127.0.98.1:41175)
0926 14:44:43.288876 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:43.288881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:43.288896 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:44:43.289068 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:43.289070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:43.289124 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:43.289205 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:44:43.289448 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.289968 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.291009 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.291232 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.291317 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.291651 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:44:43.292094 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:44:43.292196 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:43.292288 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 14:44:43.292345 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:44:43.295615 1808 raft_consensus.cc:1261] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699 [term 1 FOLLOWER]: Refusing update from remote peer 86b74a5f092b45a0b433455848712d07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:43.295782 1948 raft_consensus.cc:1261] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2 [term 1 FOLLOWER]: Refusing update from remote peer 86b74a5f092b45a0b433455848712d07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:43.296366 2024 consensus_queue.cc:1035] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:43.296752 2024 consensus_queue.cc:1035] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:43.298444 2035 tablet_replica.cc:439] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2: TxnStatusTablet state changed. Reason: New leader 86b74a5f092b45a0b433455848712d07. Latest consensus state: current_term: 1 leader_uuid: "86b74a5f092b45a0b433455848712d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } } }
I20260926 14:44:43.298561 2035 tablet_replica.cc:442] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:43.300025 2025 tablet_replica.cc:439] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699: TxnStatusTablet state changed. Reason: New leader 86b74a5f092b45a0b433455848712d07. Latest consensus state: current_term: 1 leader_uuid: "86b74a5f092b45a0b433455848712d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } } }
I20260926 14:44:43.300146 2025 tablet_replica.cc:442] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:43.300395 2024 tablet_replica.cc:439] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86b74a5f092b45a0b433455848712d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } } }
I20260926 14:44:43.300484 2024 tablet_replica.cc:442] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:44:43.304473 2025 tablet_replica.cc:439] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86b74a5f092b45a0b433455848712d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } } }
I20260926 14:44:43.304625 2025 tablet_replica.cc:442] T d7b8379a0fe04390a3ba70e0489bb03a P d5fce677d66f452a865be59b7512a699: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:43.308604 2048 tablet_replica.cc:439] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "86b74a5f092b45a0b433455848712d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } } }
I20260926 14:44:43.308740 2048 tablet_replica.cc:442] T d7b8379a0fe04390a3ba70e0489bb03a P 86b74a5f092b45a0b433455848712d07: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:44:43.309808 2035 tablet_replica.cc:439] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "86b74a5f092b45a0b433455848712d07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86b74a5f092b45a0b433455848712d07" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41175 } } peers { permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 } } peers { permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 } } }
I20260926 14:44:43.309945 2035 tablet_replica.cc:442] T d7b8379a0fe04390a3ba70e0489bb03a P f6c56d22c01a423d8a201fbfd88391a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:43.310379 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.306420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.0.98.1:41175)
0926 14:44:43.306519 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:43.306524 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:43.306577 (+ 53us) server_negotiation.cc:407] Connection header received
0926 14:44:43.306717 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:43.306720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:43.306790 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:43.306850 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:44:43.307192 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.307822 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.308864 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.309155 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.309242 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.309635 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:44:43.309737 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:44:43.309801 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:43.310203 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 14:44:43.310261 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:44:43.321671 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.318369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41175 (local address 127.0.98.1:37513)
0926 14:44:43.318704 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:43.318718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:43.318844 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:43.319037 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:43.319040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:43.319046 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:44:43.319165 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:43.319170 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.319890 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.319893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:43.320539 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:43.320547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.320964 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.320967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:43.321071 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.321112 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:44:43.321383 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:44:43.321389 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:44:43.321445 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:44:43.321541 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":21,"thread_start_us":111,"threads_started":1}
I20260926 14:44:43.321671 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.318382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37513 (local address 127.0.98.1:41175)
0926 14:44:43.318682 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:43.318686 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:43.318798 (+ 112us) server_negotiation.cc:407] Connection header received
0926 14:44:43.318927 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:43.318930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:43.318969 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:43.319026 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:44:43.319248 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.319796 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.320644 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.320870 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.320953 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.321239 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:44:43.321314 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:44:43.321369 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:43.321453 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 14:44:43.321524 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":234}
I20260926 14:44:43.328855 1808 raft_consensus.cc:1261] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699 [term 1 FOLLOWER]: Refusing update from remote peer 86b74a5f092b45a0b433455848712d07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:43.328855 1948 raft_consensus.cc:1261] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2 [term 1 FOLLOWER]: Refusing update from remote peer 86b74a5f092b45a0b433455848712d07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:43.329214 2024 consensus_queue.cc:1035] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6c56d22c01a423d8a201fbfd88391a2" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:43.329381 2068 consensus_queue.cc:1035] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5fce677d66f452a865be59b7512a699" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:43.366179 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:43.366329 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:43.368328 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90a530c4d9bb4117ac3ab6b1529cb250: Sending DeleteTablet for 3 replicas of tablet c733ff94643c4b78b44ed4623821a3e5
I20260926 14:44:43.369071 1788 tablet_service.cc:1514] Processing DeleteTablet for tablet c733ff94643c4b78b44ed4623821a3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42290
I20260926 14:44:43.369249 2040 tablet_replica.cc:330] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699: stopping tablet replica
I20260926 14:44:43.369364 2040 raft_consensus.cc:2229] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:43.369485 2040 raft_consensus.cc:2258] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:43.369709 1928 tablet_service.cc:1514] Processing DeleteTablet for tablet c733ff94643c4b78b44ed4623821a3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34504
I20260926 14:44:43.369935 2042 tablet_replica.cc:330] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2: stopping tablet replica
I20260926 14:44:43.370065 2042 raft_consensus.cc:2229] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:43.370074 1647 tablet_service.cc:1514] Processing DeleteTablet for tablet c733ff94643c4b78b44ed4623821a3e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53268
I20260926 14:44:43.370182 2042 raft_consensus.cc:2258] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:43.370270 2041 tablet_replica.cc:330] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07: stopping tablet replica
I20260926 14:44:43.370384 2041 raft_consensus.cc:2229] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:44:43.370676 2041 raft_consensus.cc:2258] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:43.370959 2042 ts_tablet_manager.cc:1905] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:43.371532 2041 ts_tablet_manager.cc:1905] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:43.373811 2042 ts_tablet_manager.cc:1918] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:44:43.373885 2042 log.cc:1196] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/c733ff94643c4b78b44ed4623821a3e5
I20260926 14:44:43.374119 2042 ts_tablet_manager.cc:1939] T c733ff94643c4b78b44ed4623821a3e5 P f6c56d22c01a423d8a201fbfd88391a2: Deleting consensus metadata
I20260926 14:44:43.374146 2041 ts_tablet_manager.cc:1918] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:44:43.374230 2041 log.cc:1196] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-0/wal/wals/c733ff94643c4b78b44ed4623821a3e5
I20260926 14:44:43.374425 2041 ts_tablet_manager.cc:1939] T c733ff94643c4b78b44ed4623821a3e5 P 86b74a5f092b45a0b433455848712d07: Deleting consensus metadata
I20260926 14:44:43.375214 2040 ts_tablet_manager.cc:1905] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:43.375370 1498 catalog_manager.cc:4841] TS 86b74a5f092b45a0b433455848712d07 (127.0.98.1:41175): tablet c733ff94643c4b78b44ed4623821a3e5 (table test-table [id=f0eb645f5d4740af8c0ef600c0bb3e0d]) successfully deleted
I20260926 14:44:43.376533 1497 catalog_manager.cc:4841] TS f6c56d22c01a423d8a201fbfd88391a2 (127.0.98.3:33145): tablet c733ff94643c4b78b44ed4623821a3e5 (table test-table [id=f0eb645f5d4740af8c0ef600c0bb3e0d]) successfully deleted
I20260926 14:44:43.377669 2040 ts_tablet_manager.cc:1918] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:44:43.377748 2040 log.cc:1196] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/c733ff94643c4b78b44ed4623821a3e5
I20260926 14:44:43.377948 2040 ts_tablet_manager.cc:1939] T c733ff94643c4b78b44ed4623821a3e5 P d5fce677d66f452a865be59b7512a699: Deleting consensus metadata
I20260926 14:44:43.378578 1497 catalog_manager.cc:4841] TS d5fce677d66f452a865be59b7512a699 (127.0.98.2:45853): tablet c733ff94643c4b78b44ed4623821a3e5 (table test-table [id=f0eb645f5d4740af8c0ef600c0bb3e0d]) successfully deleted
Sep 26 14:44:43 dist-test-slave-s7k6 krb5kdc[1466](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433882, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 14:44:43.385490 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.374691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.0.98.1:41175)
0926 14:44:43.374853 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:43.374857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:43.375120 (+ 263us) server_negotiation.cc:407] Connection header received
0926 14:44:43.375300 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:43.375303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:43.375347 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:43.375392 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:43.376411 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.377172 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.378302 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.378604 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.378709 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.382207 (+ 3498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:43.382216 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:43.382226 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:43.382251 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:43.383789 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:43.384257 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:43.384262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:43.384263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:43.384347 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:43.384608 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:43.384610 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:43.384611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:43.384859 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:43.385038 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:43.385230 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 14:44:43.385336 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260926 14:44:43.386011 1708 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53290
I20260926 14:44:43.395768 1560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:43.387728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.0.98.62:44975)
0926 14:44:43.387848 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:43.387851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:43.388050 (+ 199us) server_negotiation.cc:407] Connection header received
0926 14:44:43.388126 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:43.388129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:43.388183 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:43.388271 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:43.388847 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.389675 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.390452 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:43.390876 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:43.391015 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:43.392613 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:43.392617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:43.392636 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:43.392668 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:43.394159 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:43.394551 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:43.394554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:43.394556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:43.394639 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:43.394909 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:43.394912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:43.394914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:43.395102 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:43.395216 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:43.395482 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 14:44:43.395601 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 14:44:43.396126 1508 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39078
I20260926 14:44:43.397373 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 1583
I20260926 14:44:43.404521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 1723
I20260926 14:44:43.409369 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 1863
I20260926 14:44:43.414722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 1479
2026-09-26T14:44:43Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4216 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:44:43 dist-test-slave-s7k6 krb5kdc[2078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:44:43 dist-test-slave-s7k6 krb5kdc[2078](info): set up 2 sockets
Sep 26 14:44:43 dist-test-slave-s7k6 krb5kdc[2078](info): commencing operation
krb5kdc: starting...
W20260926 14:44:44.374401 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:44:44 dist-test-slave-s7k6 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:44:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:44:44.490878 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:44.601143 2091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:44.601426 2091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:44.601466 2091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:44.605459 2091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:44:44.605527 2091 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:44:44.605543 2091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:44.605561 2091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:44.605576 2091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:44:44.605594 2091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:44:44.610981 2091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46201
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:44.611984 2091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:44.612962 2091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:44.621650 2097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:44.621658 2096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:44.621658 2091 server_base.cc:1034] running on GCE node
W20260926 14:44:44.621675 2099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:44.622596 2091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:44.623787 2091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:44.624969 2091 hybrid_clock.cc:648] HybridClock initialized: now 1790433884624944 us; error 41 us; skew 500 ppm
Sep 26 14:44:44 dist-test-slave-s7k6 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:44.633513 2091 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:44:44.634588 2091 webserver.cc:460] Webserver started at http://127.0.98.62:45849/ using document root <none> and password file <none>
I20260926 14:44:44.635214 2091 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:44.635281 2091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:44.635491 2091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:44.637666 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "d83951ff50414297ba59687f4c76d073"
format_stamp: "Formatted at 2026-09-26 14:44:44 on dist-test-slave-s7k6"
server_key: "69d6dfcd1f28736fe1989c46f2475d4e"
server_key_iv: "da08888d500d36998454c6115909d337"
server_key_version: "encryptionkey@0"
I20260926 14:44:44.638231 2091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "d83951ff50414297ba59687f4c76d073"
format_stamp: "Formatted at 2026-09-26 14:44:44 on dist-test-slave-s7k6"
server_key: "69d6dfcd1f28736fe1989c46f2475d4e"
server_key_iv: "da08888d500d36998454c6115909d337"
server_key_version: "encryptionkey@0"
I20260926 14:44:44.642143 2091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:44:44.644688 2106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:44.645990 2091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:44:44.646111 2091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "d83951ff50414297ba59687f4c76d073"
format_stamp: "Formatted at 2026-09-26 14:44:44 on dist-test-slave-s7k6"
server_key: "69d6dfcd1f28736fe1989c46f2475d4e"
server_key_iv: "da08888d500d36998454c6115909d337"
server_key_version: "encryptionkey@0"
I20260926 14:44:44.646216 2091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:44.664378 2091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:44.667686 2091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:44.667886 2091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:44.676378 2091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46201
I20260926 14:44:44.676383 2168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46201 every 8 connection(s)
I20260926 14:44:44.677428 2091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:44:44.680356 2169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:44.685828 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073: Bootstrap starting.
I20260926 14:44:44.687011 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 2091
I20260926 14:44:44.687119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:44:44.687362 392 external_mini_cluster.cc:1428] Setting key 43fcf5e735025945cbb2b66cd86d7764
I20260926 14:44:44.688472 2169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:44.689203 2169 log.cc:824] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:44.691303 2169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073: No bootstrap required, opened a new log
I20260926 14:44:44.695312 2169 raft_consensus.cc:348] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83951ff50414297ba59687f4c76d073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
I20260926 14:44:44.695631 2169 raft_consensus.cc:374] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:44.695694 2169 raft_consensus.cc:729] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d83951ff50414297ba59687f4c76d073, State: Initialized, Role: FOLLOWER
I20260926 14:44:44.696147 2169 consensus_queue.cc:260] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83951ff50414297ba59687f4c76d073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
Sep 26 14:44:44 dist-test-slave-s7k6 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:44.696280 2169 raft_consensus.cc:388] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:44:44.696346 2169 raft_consensus.cc:482] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:44:44.696448 2169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:44.697337 2169 raft_consensus.cc:504] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83951ff50414297ba59687f4c76d073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
I20260926 14:44:44.697664 2169 leader_election.cc:302] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d83951ff50414297ba59687f4c76d073; no voters:
I20260926 14:44:44.697983 2169 leader_election.cc:288] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:44:44.698230 2174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:44.698457 2174 raft_consensus.cc:686] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [term 1 LEADER]: Becoming Leader. State: Replica: d83951ff50414297ba59687f4c76d073, State: Running, Role: LEADER
I20260926 14:44:44.698850 2174 consensus_queue.cc:237] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83951ff50414297ba59687f4c76d073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } }
I20260926 14:44:44.700160 2174 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d83951ff50414297ba59687f4c76d073. Latest consensus state: current_term: 1 leader_uuid: "d83951ff50414297ba59687f4c76d073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83951ff50414297ba59687f4c76d073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } } }
I20260926 14:44:44.700932 2174 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:44.700793 2175 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d83951ff50414297ba59687f4c76d073" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d83951ff50414297ba59687f4c76d073" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46201 } } }
I20260926 14:44:44.701248 2175 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:44.702657 2169 sys_catalog.cc:564] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:44:44.702858 2178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:44:44.702988 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:44.688391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38184 (local address 127.0.98.62:46201)
0926 14:44:44.688886 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:44.688900 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:44.688934 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:44:44.689581 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:44.689600 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:44.689939 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:44.690233 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:44.691132 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:44.692209 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:44.693420 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:44.693950 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:44.694080 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:44.696853 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:44.696858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:44.696872 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:44.696938 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:44.698976 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:44.699613 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:44.699619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:44.699626 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:44.699781 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:44.700111 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:44.700114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:44.700116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:44.700619 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:44.701441 (+ 822us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:44.701615 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 14:44:44.701945 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":130,"threads_started":1}
I20260926 14:44:44.706197 2178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:44:44.712345 2178 catalog_manager.cc:1347] Generated new cluster ID: 0579fb15052f42fda3c5efd1558cd1de
I20260926 14:44:44.712415 2178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:44:44.745800 2178 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:44:44.747252 2178 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:44:44.771116 2178 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073: Generated new TSK 0
I20260926 14:44:44.772053 2178 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:44:44.805858 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46201
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:44.916460 2195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:44.916739 2195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:44.916791 2195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:44.921160 2195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:44.921226 2195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:44.921268 2195 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:44:44.921303 2195 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:44:44.921355 2195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:44:44.926826 2195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:44.927922 2195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:44.928925 2195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:44.938335 2200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:44.938406 2201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:44.938352 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:44.938575 2195 server_base.cc:1034] running on GCE node
I20260926 14:44:44.939064 2195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:44.939713 2195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:44.940901 2195 hybrid_clock.cc:648] HybridClock initialized: now 1790433884940880 us; error 41 us; skew 500 ppm
Sep 26 14:44:44 dist-test-slave-s7k6 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:44.943821 2195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:44:44.944759 2195 webserver.cc:460] Webserver started at http://127.0.98.1:42509/ using document root <none> and password file <none>
I20260926 14:44:44.945312 2195 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:44.945405 2195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:44.945609 2195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:44.947781 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "8a2f6265b49e4e58955ce05227fe108f"
format_stamp: "Formatted at 2026-09-26 14:44:44 on dist-test-slave-s7k6"
server_key: "dc818afbfff9547aef3a0f4739a3bffd"
server_key_iv: "0f56a000334a969ae9922814258c7660"
server_key_version: "encryptionkey@0"
I20260926 14:44:44.948359 2195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8a2f6265b49e4e58955ce05227fe108f"
format_stamp: "Formatted at 2026-09-26 14:44:44 on dist-test-slave-s7k6"
server_key: "dc818afbfff9547aef3a0f4739a3bffd"
server_key_iv: "0f56a000334a969ae9922814258c7660"
server_key_version: "encryptionkey@0"
I20260926 14:44:44.952205 2195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:44:44.954717 2210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:44.956110 2195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:44:44.956267 2195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "8a2f6265b49e4e58955ce05227fe108f"
format_stamp: "Formatted at 2026-09-26 14:44:44 on dist-test-slave-s7k6"
server_key: "dc818afbfff9547aef3a0f4739a3bffd"
server_key_iv: "0f56a000334a969ae9922814258c7660"
server_key_version: "encryptionkey@0"
I20260926 14:44:44.956373 2195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:44.971872 2195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:44.975063 2195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:44.975238 2195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:44.976679 2195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:44.976728 2195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:44.976783 2195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:44.976809 2195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:44:44 dist-test-slave-s7k6 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:44.986843 2195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41933
I20260926 14:44:44.987011 2325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41933 every 8 connection(s)
I20260926 14:44:44.987838 2195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:44:44.990056 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:44.977470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55293 (local address 127.0.98.62:46201)
0926 14:44:44.977581 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:44.977584 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:44.978657 (+ 1073us) server_negotiation.cc:407] Connection header received
0926 14:44:44.979742 (+ 1085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:44.979745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:44.979817 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:44.979895 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:44.981514 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:44.982356 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:44.983715 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:44.983952 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:44.984054 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:44.986722 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:44.986726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:44.986729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:44.986795 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:44.988286 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:44.988821 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:44.988824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:44.988826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:44.988909 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:44.989154 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:44.989156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:44.989157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:44.989323 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:44.989392 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:44.989825 (+ 433us) server_negotiation.cc:299] Negotiation successful
0926 14:44:44.989935 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:44:44.990944 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:44.977991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.1:55293)
0926 14:44:44.978498 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:44.978541 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:44.979511 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:44.980205 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:44.980215 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:44.980647 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:44.981332 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:44.981354 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:44.982502 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:44.982506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:44.983589 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:44.983599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:44.984136 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:44.984140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:44.984320 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:44.984948 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:44.984969 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:44.986484 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:44.988431 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:44.988436 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:44.988452 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:44.988739 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:44.988978 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:44.988980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:44.988982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:44.989100 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:44.989393 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:44.989397 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:44.989635 (+ 238us) client_negotiation.cc:769] Sending connection context
0926 14:44:44.989789 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 14:44:44.990046 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":121,"threads_started":1}
I20260926 14:44:44.991887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 2195
I20260926 14:44:44.992007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:44:44.992331 392 external_mini_cluster.cc:1428] Setting key f6aba0d1d5d37e50c510256d138995d7
I20260926 14:44:44.992352 2326 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46201
I20260926 14:44:44.992525 2326 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:44.993026 2326 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
I20260926 14:44:44.994572 2122 ts_manager.cc:194] Registered new tserver with Master: 8a2f6265b49e4e58955ce05227fe108f (127.0.98.1:41933)
I20260926 14:44:44.995785 2122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55293
I20260926 14:44:45.003366 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:44.994574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.1:39295)
0926 14:44:44.996479 (+ 1905us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:44.996491 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:44.996581 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:44.996800 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:44.996803 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:44.997000 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:44.997096 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:44.997102 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:44.997898 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:44.997901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:44.998455 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:44.998463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:44.998926 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:44.998930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:44.999016 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:44.999498 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:44.999510 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:44.999841 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:45.001737 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.001740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.001741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.001945 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.002261 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.002264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.002266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.002352 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.002849 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:45.002851 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:45.002938 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.003030 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.003176 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1817,"thread_start_us":88,"threads_started":1}
I20260926 14:44:45.003386 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:44.994646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39295 (local address 127.0.98.62:46201)
0926 14:44:44.994782 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:44.994786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:44.996552 (+ 1766us) server_negotiation.cc:407] Connection header received
0926 14:44:44.996659 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:44.996662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:44.996703 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:44.996762 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:44.997219 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:44.997774 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:44.998559 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:44.998822 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:44.998911 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:44.999987 (+ 1076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:44.999991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:44.999994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:45.000019 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:45.001621 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.002049 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.002052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.002054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.002144 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.002496 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.002499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.002501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.002711 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:45.002867 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.003107 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.003227 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:44:45.021687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46201
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:45.144382 2335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:45.144675 2335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:45.144727 2335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:45.149057 2335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:45.149127 2335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:45.149168 2335 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:44:45.149183 2335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:44:45.149236 2335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:44:45.154913 2335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:45.156029 2335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:45.157074 2335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:45.166921 2340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:45.166911 2341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:45.166908 2343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:45.166994 2335 server_base.cc:1034] running on GCE node
I20260926 14:44:45.167557 2335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:45.168205 2335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:45.169394 2335 hybrid_clock.cc:648] HybridClock initialized: now 1790433885169344 us; error 63 us; skew 500 ppm
Sep 26 14:44:45 dist-test-slave-s7k6 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:45.172200 2335 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:44:45.173082 2335 webserver.cc:460] Webserver started at http://127.0.98.2:41963/ using document root <none> and password file <none>
I20260926 14:44:45.173614 2335 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:45.173669 2335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:45.173831 2335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:45.175912 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "21fe479a9cc3473db1175eec1489afcf"
format_stamp: "Formatted at 2026-09-26 14:44:45 on dist-test-slave-s7k6"
server_key: "f0f51932c3faee37c4afb78c96838e75"
server_key_iv: "d9ab508ebdc0f10bc129d4fc33a82a28"
server_key_version: "encryptionkey@0"
I20260926 14:44:45.176442 2335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "21fe479a9cc3473db1175eec1489afcf"
format_stamp: "Formatted at 2026-09-26 14:44:45 on dist-test-slave-s7k6"
server_key: "f0f51932c3faee37c4afb78c96838e75"
server_key_iv: "d9ab508ebdc0f10bc129d4fc33a82a28"
server_key_version: "encryptionkey@0"
I20260926 14:44:45.180172 2335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:44:45.182655 2350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:45.184041 2335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:44:45.184150 2335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "21fe479a9cc3473db1175eec1489afcf"
format_stamp: "Formatted at 2026-09-26 14:44:45 on dist-test-slave-s7k6"
server_key: "f0f51932c3faee37c4afb78c96838e75"
server_key_iv: "d9ab508ebdc0f10bc129d4fc33a82a28"
server_key_version: "encryptionkey@0"
I20260926 14:44:45.184249 2335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:45.199182 2335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:45.202550 2335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:45.202718 2335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:45.204433 2335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:45.204496 2335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:45.204536 2335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:45.204553 2335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:44:45 dist-test-slave-s7k6 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:45.214353 2335 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38017
I20260926 14:44:45.214404 2465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38017 every 8 connection(s)
I20260926 14:44:45.215267 2335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:44:45.217453 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.205143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45335 (local address 127.0.98.62:46201)
0926 14:44:45.205280 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.205284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.206332 (+ 1048us) server_negotiation.cc:407] Connection header received
0926 14:44:45.207411 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.207415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.207473 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.207580 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:45.208880 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.209688 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.211115 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.211340 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.211432 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.213870 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:45.213874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:45.213877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:45.213907 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:45.215471 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.216056 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.216059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.216061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.216148 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.216478 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.216481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.216482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.216665 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:45.216739 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.217226 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.217330 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:44:45.218240 2359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.205661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.2:45335)
0926 14:44:45.206164 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.206215 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.207217 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.207771 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.207779 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.208186 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:45.208738 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.208756 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.209814 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.209818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.210999 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.211009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.211499 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.211502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.211673 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.212223 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:45.212243 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:45.213691 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:45.215581 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.215587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.215601 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.215934 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.216240 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.216242 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.216244 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.216383 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.216742 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:45.216748 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:45.217026 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.217186 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.217446 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":95,"threads_started":1}
I20260926 14:44:45.219063 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 2335
I20260926 14:44:45.219164 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:44:45.219408 392 external_mini_cluster.cc:1428] Setting key dadf3318e9d0c41dee859da6bca9a45f
I20260926 14:44:45.219583 2466 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46201
I20260926 14:44:45.219723 2466 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:45.220237 2466 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
I20260926 14:44:45.223040 2123 ts_manager.cc:194] Registered new tserver with Master: 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017)
I20260926 14:44:45.223619 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45335
I20260926 14:44:45.229507 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.221795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44303 (local address 127.0.98.62:46201)
0926 14:44:45.221910 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.221914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.222173 (+ 259us) server_negotiation.cc:407] Connection header received
0926 14:44:45.222265 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.222267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.222307 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.222384 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:45.222919 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.223654 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.224790 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.225046 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.225133 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.226351 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:45.226354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:45.226356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:45.226378 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:45.227803 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.228320 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.228324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.228325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.228423 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.228707 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.228709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.228710 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.228926 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:45.229003 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.229267 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.229367 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:44:45.229588 2472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.221692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.2:44303)
0926 14:44:45.222072 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.222086 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.222185 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.222410 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.222412 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.222628 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:45.222748 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.222773 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.223794 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.223798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.224655 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.224667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.225160 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.225164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.225252 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.225754 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:45.225768 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:45.226204 (+ 436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:45.227918 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.227922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.227925 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.228200 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.228519 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.228523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.228525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.228622 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.229014 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:45.229016 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:45.229118 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.229232 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.229386 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:44:45.247484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46201
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:45.358043 2475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:45.358318 2475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:45.358366 2475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:45.362522 2475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:45.362588 2475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:45.362628 2475 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 14:44:45.362641 2475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:44:45.362685 2475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:44:45.368198 2475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46201
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:45.369206 2475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:45.370189 2475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:45.379858 2480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:45.379868 2483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:45.379946 2475 server_base.cc:1034] running on GCE node
W20260926 14:44:45.379863 2481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:45.380541 2475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:45.381201 2475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:45.382370 2475 hybrid_clock.cc:648] HybridClock initialized: now 1790433885382346 us; error 38 us; skew 500 ppm
Sep 26 14:44:45 dist-test-slave-s7k6 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:45.385082 2475 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:44:45.385978 2475 webserver.cc:460] Webserver started at http://127.0.98.3:43139/ using document root <none> and password file <none>
I20260926 14:44:45.386565 2475 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:45.386637 2475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:45.386886 2475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:45.389065 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "d32454af21284b218fb244f64b439914"
format_stamp: "Formatted at 2026-09-26 14:44:45 on dist-test-slave-s7k6"
server_key: "957de7931e8dd889950c1d61627e9649"
server_key_iv: "2d73d8f6c27fa0c270441760bdc2c993"
server_key_version: "encryptionkey@0"
I20260926 14:44:45.389601 2475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d32454af21284b218fb244f64b439914"
format_stamp: "Formatted at 2026-09-26 14:44:45 on dist-test-slave-s7k6"
server_key: "957de7931e8dd889950c1d61627e9649"
server_key_iv: "2d73d8f6c27fa0c270441760bdc2c993"
server_key_version: "encryptionkey@0"
I20260926 14:44:45.393468 2475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:44:45.396178 2490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:45.397763 2475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:44:45.397986 2475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "d32454af21284b218fb244f64b439914"
format_stamp: "Formatted at 2026-09-26 14:44:45 on dist-test-slave-s7k6"
server_key: "957de7931e8dd889950c1d61627e9649"
server_key_iv: "2d73d8f6c27fa0c270441760bdc2c993"
server_key_version: "encryptionkey@0"
I20260926 14:44:45.398114 2475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:45.430017 2475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:45.433213 2475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:45.433410 2475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:45.434996 2475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:45.435048 2475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:45.435101 2475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:45.435127 2475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:44:45 dist-test-slave-s7k6 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:45.445067 2475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44305
I20260926 14:44:45.445094 2605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44305 every 8 connection(s)
I20260926 14:44:45.446136 2475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:44:45.448835 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.435857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37259 (local address 127.0.98.62:46201)
0926 14:44:45.435994 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.436000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.436958 (+ 958us) server_negotiation.cc:407] Connection header received
0926 14:44:45.438091 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.438095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.438152 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.438240 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:45.439943 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.440568 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.441787 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.442024 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.442116 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.445045 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:45.445049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:45.445051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:45.445078 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:45.446572 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.447254 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.447257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.447259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.447362 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.447739 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.447742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.447743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.447991 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:45.448074 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.448595 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.448716 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:44:45.449684 2499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.436286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.3:37259)
0926 14:44:45.436780 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.436831 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.437821 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.438478 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.438489 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.439023 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:45.439743 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.439765 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.440696 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.440699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.441669 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.441680 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.442166 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.442170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.442330 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.443036 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:45.443062 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:45.444840 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:45.446714 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.446719 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.446733 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.447128 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.447473 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.447475 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.447477 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.447613 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.448100 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:45.448105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:45.448350 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.448501 (+ 151us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.448757 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260926 14:44:45.451103 2606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46201
I20260926 14:44:45.451256 2606 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:45.451786 2606 heartbeater.cc:511] Master 127.0.98.62:46201 requested a full tablet report, sending...
I20260926 14:44:45.452898 2123 ts_manager.cc:194] Registered new tserver with Master: d32454af21284b218fb244f64b439914 (127.0.98.3:44305)
I20260926 14:44:45.453615 2123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37259
I20260926 14:44:45.453687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 2475
I20260926 14:44:45.453800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:44:45.454133 392 external_mini_cluster.cc:1428] Setting key bf57cdb934a7f2a3bf26374b4854bc63
I20260926 14:44:45.458063 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:44:45.461182 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.453319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37521 (local address 127.0.98.62:46201)
0926 14:44:45.453424 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.453429 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.453693 (+ 264us) server_negotiation.cc:407] Connection header received
0926 14:44:45.453799 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.453802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.453845 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.453925 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:45.454435 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.455223 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.456293 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.456597 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.456703 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.457965 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:45.457969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:45.457972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:45.457998 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:45.459379 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.459870 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.459874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.459875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.459976 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.460311 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.460314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.460315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.460494 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:45.460568 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.460903 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.461028 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 14:44:45.461658 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.453263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.98.3:37521)
0926 14:44:45.453593 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.453608 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.453718 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.453957 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.453961 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.454172 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:45.454298 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.454306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.455380 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.455383 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.456197 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.456207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.456737 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.456740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.456838 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.457425 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:45.457441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:45.457837 (+ 396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:45.459496 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.459500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.459502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.459770 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.460113 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.460117 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.460118 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.460216 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.460585 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:45.460589 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:45.460707 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.461317 (+ 610us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.461484 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":123,"threads_started":1}
Sep 26 14:44:45 dist-test-slave-s7k6 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:44:45 dist-test-slave-s7k6 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:45.483539 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.475639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35080 (local address 127.0.98.62:46201)
0926 14:44:45.475766 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.475770 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.475781 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:44:45.475820 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.475822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.475860 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.475927 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:45.476613 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.477128 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.477948 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.478159 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.478245 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.480620 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:45.480623 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:45.480625 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:45.480651 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:45.481917 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.482310 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.482314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.482316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.482396 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.482637 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.482640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.482642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.482863 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:45.482981 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.483261 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.483373 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:44:45.486519 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:44:45.489671 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:44:45.504812 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.498616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37006 (local address 127.0.98.2:38017)
0926 14:44:45.499457 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.499469 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.499491 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:44:45.499562 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.499566 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.499786 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.500234 (+ 448us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:45.500356 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.501520 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.503139 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.503989 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.504358 (+ 369us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.504509 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.504559 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.504658 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":91,"threads_started":1}
I20260926 14:44:45.504802 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.499489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48226 (local address 127.0.98.1:41933)
0926 14:44:45.499839 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.499846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.500161 (+ 315us) server_negotiation.cc:407] Connection header received
0926 14:44:45.500238 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.500243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.500425 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.500566 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:45.500736 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.501779 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.503206 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.503999 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.504074 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.504212 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.504561 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.504658 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":56,"threads_started":1}
I20260926 14:44:45.504807 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.498476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38017 (local address 127.0.0.1:37006)
0926 14:44:45.498973 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.499024 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.499160 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.499950 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.499957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.499979 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:45.500092 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.500097 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.501624 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.501627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.503029 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.503039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.504155 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.504158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.504280 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.504405 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.504472 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.504654 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"mutex_wait_us":186,"spinlock_wait_cycles":6784,"thread_start_us":99,"threads_started":1}
I20260926 14:44:45.505283 2626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.499243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41933 (local address 127.0.0.1:48226)
0926 14:44:45.500049 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.500061 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.500152 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.500559 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.500561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.500577 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:45.500651 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.500656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.501881 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.501885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.503075 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.503084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.504312 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.504315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.504410 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.504429 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.505055 (+ 626us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.505125 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260926 14:44:45.505313 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.498607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44305 (local address 127.0.0.1:41378)
0926 14:44:45.499011 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.499025 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.499159 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.500324 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.500327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.500351 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:45.500442 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.500448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.501513 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.501517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.503125 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.503132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.504234 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.504237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.504303 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.504350 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.504492 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.505168 (+ 676us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"mutex_wait_us":649,"spinlock_wait_cycles":3840,"thread_start_us":111,"threads_started":1}
I20260926 14:44:45.505584 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.499538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.0.98.3:44305)
0926 14:44:45.499897 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.499908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.499929 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:44:45.500002 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.500007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.500203 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.500343 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:45.500544 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.501363 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.503225 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.503915 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.504015 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.504182 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.504687 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.504782 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":95,"threads_started":1}
I20260926 14:44:45.507985 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 60b83540a10745299d3e50004045919f (DEFAULT_TABLE table=test-table [id=f39b414afc984a75a5203174c854cc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:45.508203 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 60b83540a10745299d3e50004045919f (DEFAULT_TABLE table=test-table [id=f39b414afc984a75a5203174c854cc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:45.508201 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 60b83540a10745299d3e50004045919f (DEFAULT_TABLE table=test-table [id=f39b414afc984a75a5203174c854cc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:45.509020 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b83540a10745299d3e50004045919f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:45.509203 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b83540a10745299d3e50004045919f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:45.509209 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60b83540a10745299d3e50004045919f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:45.515811 2629 tablet_bootstrap.cc:492] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf: Bootstrap starting.
I20260926 14:44:45.517074 2630 tablet_bootstrap.cc:492] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: Bootstrap starting.
I20260926 14:44:45.518193 2629 tablet_bootstrap.cc:654] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:45.518955 2630 tablet_bootstrap.cc:654] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:45.519042 2629 log.cc:824] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:45.519644 2630 log.cc:824] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:45.520279 2631 tablet_bootstrap.cc:492] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914: Bootstrap starting.
I20260926 14:44:45.521399 2629 tablet_bootstrap.cc:492] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf: No bootstrap required, opened a new log
I20260926 14:44:45.521564 2630 tablet_bootstrap.cc:492] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: No bootstrap required, opened a new log
I20260926 14:44:45.521589 2629 ts_tablet_manager.cc:1397] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:44:45.521751 2630 ts_tablet_manager.cc:1397] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260926 14:44:45.522437 2631 tablet_bootstrap.cc:654] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:45.523260 2631 log.cc:824] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:45.524834 2629 raft_consensus.cc:348] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } }
I20260926 14:44:45.525077 2629 raft_consensus.cc:374] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:45.525130 2629 raft_consensus.cc:729] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21fe479a9cc3473db1175eec1489afcf, State: Initialized, Role: FOLLOWER
I20260926 14:44:45.525350 2631 tablet_bootstrap.cc:492] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914: No bootstrap required, opened a new log
I20260926 14:44:45.525554 2631 ts_tablet_manager.cc:1397] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:44:45.525568 2629 consensus_queue.cc:260] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } }
I20260926 14:44:45.525782 2630 raft_consensus.cc:348] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } }
I20260926 14:44:45.526055 2630 raft_consensus.cc:374] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:45.526129 2630 raft_consensus.cc:729] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2f6265b49e4e58955ce05227fe108f, State: Initialized, Role: FOLLOWER
I20260926 14:44:45.526355 2466 heartbeater.cc:503] Master 127.0.98.62:46201 was elected leader, sending a full tablet report...
I20260926 14:44:45.526633 2629 ts_tablet_manager.cc:1428] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:44:45.526607 2630 consensus_queue.cc:260] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } }
I20260926 14:44:45.527562 2630 ts_tablet_manager.cc:1428] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:44:45.527928 2326 heartbeater.cc:503] Master 127.0.98.62:46201 was elected leader, sending a full tablet report...
I20260926 14:44:45.529196 2631 raft_consensus.cc:348] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } }
I20260926 14:44:45.529438 2631 raft_consensus.cc:374] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:45.529507 2631 raft_consensus.cc:729] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32454af21284b218fb244f64b439914, State: Initialized, Role: FOLLOWER
I20260926 14:44:45.530000 2631 consensus_queue.cc:260] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } }
I20260926 14:44:45.530648 2606 heartbeater.cc:503] Master 127.0.98.62:46201 was elected leader, sending a full tablet report...
I20260926 14:44:45.531032 2631 ts_tablet_manager.cc:1428] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 14:44:45.588088 2636 raft_consensus.cc:482] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:44:45.588308 2636 raft_consensus.cc:504] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } }
I20260926 14:44:45.589546 2636 leader_election.cc:288] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017), d32454af21284b218fb244f64b439914 (127.0.98.3:44305)
I20260926 14:44:45.595105 2639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.589797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38017 (local address 127.0.98.1:56583)
0926 14:44:45.590202 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.590217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.590366 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.590671 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.590675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.590793 (+ 118us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:45.591046 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.591058 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.592199 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.592203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.593689 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.593702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.594605 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.594609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.594706 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.594733 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.594832 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.594917 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":58,"spinlock_wait_cycles":33024,"thread_start_us":141,"threads_started":1}
I20260926 14:44:45.595196 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.589906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56583 (local address 127.0.98.2:38017)
0926 14:44:45.590036 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.590041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.590315 (+ 274us) server_negotiation.cc:407] Connection header received
0926 14:44:45.590472 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.590475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.590529 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.590617 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:45.591219 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.592073 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.593815 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.594484 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.594604 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.594674 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.594990 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.595071 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:44:45.595198 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.590004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56639 (local address 127.0.98.3:44305)
0926 14:44:45.590129 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.590133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.590332 (+ 199us) server_negotiation.cc:407] Connection header received
0926 14:44:45.590470 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.590473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.590529 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.590616 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:45.591233 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.592035 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.593868 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.594524 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.594612 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.594674 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.595024 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.595098 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:44:45.595573 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.589849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44305 (local address 127.0.98.1:56639)
0926 14:44:45.590254 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.590265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.590366 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.590671 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.590675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.590707 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:45.591056 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.591062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.592199 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.592203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.593689 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.593702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.594738 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.594741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.594879 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.594903 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.595337 (+ 434us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.595422 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":288,"spinlock_wait_cycles":3456,"thread_start_us":116,"threads_started":1}
I20260926 14:44:45.595757 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b83540a10745299d3e50004045919f" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fe479a9cc3473db1175eec1489afcf" is_pre_election: true
I20260926 14:44:45.596097 2420 raft_consensus.cc:2454] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 0.
I20260926 14:44:45.596642 2214 leader_election.cc:302] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21fe479a9cc3473db1175eec1489afcf, 8a2f6265b49e4e58955ce05227fe108f; no voters:
I20260926 14:44:45.596766 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b83540a10745299d3e50004045919f" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32454af21284b218fb244f64b439914" is_pre_election: true
I20260926 14:44:45.596916 2636 raft_consensus.cc:2781] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:44:45.597002 2636 raft_consensus.cc:482] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:44:45.597033 2636 raft_consensus.cc:3037] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:45.597079 2560 raft_consensus.cc:2454] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 0.
I20260926 14:44:45.598153 2636 raft_consensus.cc:504] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } }
I20260926 14:44:45.598505 2636 leader_election.cc:288] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 election: Requested vote from peers 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017), d32454af21284b218fb244f64b439914 (127.0.98.3:44305)
I20260926 14:44:45.598820 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b83540a10745299d3e50004045919f" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fe479a9cc3473db1175eec1489afcf"
I20260926 14:44:45.598841 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60b83540a10745299d3e50004045919f" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32454af21284b218fb244f64b439914"
I20260926 14:44:45.598935 2420 raft_consensus.cc:3037] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:45.598949 2560 raft_consensus.cc:3037] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:45.599927 2560 raft_consensus.cc:2454] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 1.
I20260926 14:44:45.600204 2211 leader_election.cc:302] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a2f6265b49e4e58955ce05227fe108f, d32454af21284b218fb244f64b439914; no voters:
I20260926 14:44:45.600240 2420 raft_consensus.cc:2454] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 1.
I20260926 14:44:45.600380 2636 raft_consensus.cc:2781] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:45.600577 2636 raft_consensus.cc:686] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 1 LEADER]: Becoming Leader. State: Replica: 8a2f6265b49e4e58955ce05227fe108f, State: Running, Role: LEADER
I20260926 14:44:45.600898 2636 consensus_queue.cc:237] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } }
I20260926 14:44:45.603989 2123 catalog_manager.cc:5495] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2f6265b49e4e58955ce05227fe108f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } health_report { overall_health: HEALTHY } } }
Sep 26 14:44:45 dist-test-slave-s7k6 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:45.656968 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.648568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46201 (local address 127.0.0.1:35082)
0926 14:44:45.648885 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.648897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.649023 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.649249 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.649252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.649725 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:45.649828 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.649835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.650625 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.650629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.651267 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.651278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.651732 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.651736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.651802 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.652407 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:45.652424 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:45.653868 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:45.655537 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.655546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.655552 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.655787 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.656052 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:45.656055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:45.656056 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:45.656130 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:45.656434 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:45.656439 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:45.656572 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.656661 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.656786 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":119,"threads_started":1}
I20260926 14:44:45.656975 2172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.648649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.0.98.62:46201)
0926 14:44:45.648792 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.648796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.648971 (+ 175us) server_negotiation.cc:407] Connection header received
0926 14:44:45.649118 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.649121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.649155 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.649212 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:45.649947 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.650510 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.651388 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.651630 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.651698 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.654053 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:45.654056 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:45.654059 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:45.654080 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:45.655409 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.655870 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.655873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.655874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.655959 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:45.656196 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:45.656198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:45.656199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:45.656362 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:45.656429 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.656721 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.656841 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":52}
I20260926 14:44:45.660941 2123 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35082:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:44:45.661677 2123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:44:45.666483 2260 tablet_service.cc:1467] Processing CreateTablet for tablet 44846855f260449a9a7eb2bf8f3c9b58 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67fa05998a4e45b399e3a985e3f76466]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:44:45.666483 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 44846855f260449a9a7eb2bf8f3c9b58 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67fa05998a4e45b399e3a985e3f76466]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:44:45.666915 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 44846855f260449a9a7eb2bf8f3c9b58 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=67fa05998a4e45b399e3a985e3f76466]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:44:45.667351 2260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44846855f260449a9a7eb2bf8f3c9b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:45.667351 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44846855f260449a9a7eb2bf8f3c9b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:45.667470 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44846855f260449a9a7eb2bf8f3c9b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:45.670020 2630 tablet_bootstrap.cc:492] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: Bootstrap starting.
I20260926 14:44:45.670171 2629 tablet_bootstrap.cc:492] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: Bootstrap starting.
I20260926 14:44:45.670415 2631 tablet_bootstrap.cc:492] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: Bootstrap starting.
I20260926 14:44:45.671154 2629 tablet_bootstrap.cc:654] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:45.671223 2630 tablet_bootstrap.cc:654] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:45.671315 2631 tablet_bootstrap.cc:654] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:45.672430 2630 tablet_bootstrap.cc:492] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: No bootstrap required, opened a new log
I20260926 14:44:45.672523 2631 tablet_bootstrap.cc:492] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: No bootstrap required, opened a new log
I20260926 14:44:45.672526 2630 ts_tablet_manager.cc:1397] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 14:44:45.672606 2631 ts_tablet_manager.cc:1397] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:44:45.672708 2629 tablet_bootstrap.cc:492] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: No bootstrap required, opened a new log
I20260926 14:44:45.672793 2629 ts_tablet_manager.cc:1397] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:44:45.673018 2631 raft_consensus.cc:348] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.673007 2630 raft_consensus.cc:348] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.673128 2631 raft_consensus.cc:374] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:45.673130 2630 raft_consensus.cc:374] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:45.673163 2631 raft_consensus.cc:729] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32454af21284b218fb244f64b439914, State: Initialized, Role: FOLLOWER
I20260926 14:44:45.673169 2630 raft_consensus.cc:729] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2f6265b49e4e58955ce05227fe108f, State: Initialized, Role: FOLLOWER
I20260926 14:44:45.673285 2631 consensus_queue.cc:260] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.673231 2629 raft_consensus.cc:348] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.673363 2629 raft_consensus.cc:374] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:45.673401 2629 raft_consensus.cc:729] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21fe479a9cc3473db1175eec1489afcf, State: Initialized, Role: FOLLOWER
I20260926 14:44:45.673532 2631 ts_tablet_manager.cc:1428] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:44:45.673527 2629 consensus_queue.cc:260] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.673547 2630 consensus_queue.cc:260] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.673777 2629 ts_tablet_manager.cc:1428] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:44:45.673768 2636 tablet_replica.cc:439] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } }
I20260926 14:44:45.673786 2637 tablet_replica.cc:439] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } }
I20260926 14:44:45.673907 2636 tablet_replica.cc:442] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:45.673926 2637 tablet_replica.cc:442] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:45.674150 2630 ts_tablet_manager.cc:1428] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:44:45.674393 2635 tablet_replica.cc:439] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } }
I20260926 14:44:45.674517 2635 tablet_replica.cc:442] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:45.690539 2636 raft_consensus.cc:482] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:44:45.690653 2636 raft_consensus.cc:504] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.691068 2636 leader_election.cc:288] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017), d32454af21284b218fb244f64b439914 (127.0.98.3:44305)
I20260926 14:44:45.691428 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44846855f260449a9a7eb2bf8f3c9b58" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32454af21284b218fb244f64b439914" is_pre_election: true
I20260926 14:44:45.691433 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44846855f260449a9a7eb2bf8f3c9b58" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fe479a9cc3473db1175eec1489afcf" is_pre_election: true
I20260926 14:44:45.691579 2560 raft_consensus.cc:2454] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 0.
I20260926 14:44:45.691579 2420 raft_consensus.cc:2454] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 0.
I20260926 14:44:45.691845 2214 leader_election.cc:302] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21fe479a9cc3473db1175eec1489afcf, 8a2f6265b49e4e58955ce05227fe108f; no voters:
I20260926 14:44:45.692010 2636 raft_consensus.cc:2781] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:44:45.692065 2636 raft_consensus.cc:482] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:44:45.692095 2636 raft_consensus.cc:3037] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:45.693050 2636 raft_consensus.cc:504] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.693315 2636 leader_election.cc:288] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 election: Requested vote from peers 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017), d32454af21284b218fb244f64b439914 (127.0.98.3:44305)
I20260926 14:44:45.693679 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44846855f260449a9a7eb2bf8f3c9b58" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fe479a9cc3473db1175eec1489afcf"
I20260926 14:44:45.693780 2420 raft_consensus.cc:3037] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:45.693745 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44846855f260449a9a7eb2bf8f3c9b58" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32454af21284b218fb244f64b439914"
I20260926 14:44:45.693863 2560 raft_consensus.cc:3037] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:45.694629 2420 raft_consensus.cc:2454] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 1.
I20260926 14:44:45.694816 2560 raft_consensus.cc:2454] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 1.
I20260926 14:44:45.694913 2214 leader_election.cc:302] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21fe479a9cc3473db1175eec1489afcf, 8a2f6265b49e4e58955ce05227fe108f; no voters:
I20260926 14:44:45.695118 2636 raft_consensus.cc:2781] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:45.695212 2636 raft_consensus.cc:686] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [term 1 LEADER]: Becoming Leader. State: Replica: 8a2f6265b49e4e58955ce05227fe108f, State: Running, Role: LEADER
I20260926 14:44:45.695341 2636 consensus_queue.cc:237] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.695955 2640 tablet_replica.cc:439] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: TxnStatusTablet state changed. Reason: New leader 8a2f6265b49e4e58955ce05227fe108f. Latest consensus state: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } }
I20260926 14:44:45.696049 2640 tablet_replica.cc:442] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:44:45.696316 2652 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:44:45.696520 2652 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:44:45.696918 2122 catalog_manager.cc:5495] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2f6265b49e4e58955ce05227fe108f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } health_report { overall_health: UNKNOWN } } }
W20260926 14:44:45.697165 2607 tablet.cc:2369] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:44:45.714354 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.710804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48242 (local address 127.0.98.1:41933)
0926 14:44:45.710917 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.710921 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.711042 (+ 121us) server_negotiation.cc:407] Connection header received
0926 14:44:45.711242 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.711244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.711301 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.711385 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:44:45.711616 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.712121 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.712930 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.713152 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.713244 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.713545 (+ 301us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:44:45.713975 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:44:45.714077 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.714151 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.714207 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 14:44:45.714354 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.710647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41933 (local address 127.0.0.1:48242)
0926 14:44:45.710972 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.710985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.711095 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.711406 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.711408 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.711415 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:44:45.711515 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.711521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.712218 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.712221 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.712819 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.712826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.713253 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.713256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.713347 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.713411 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:44:45.714062 (+ 651us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:44:45.714078 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.714141 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.714230 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
W20260926 14:44:45.716483 2467 tablet.cc:2369] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:44:45.719797 2560 raft_consensus.cc:1261] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Refusing update from remote peer 8a2f6265b49e4e58955ce05227fe108f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:45.719818 2420 raft_consensus.cc:1261] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Refusing update from remote peer 8a2f6265b49e4e58955ce05227fe108f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:45.720460 2640 consensus_queue.cc:1035] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:45.720674 2636 consensus_queue.cc:1035] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:45.724390 2637 tablet_replica.cc:439] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: TxnStatusTablet state changed. Reason: New leader 8a2f6265b49e4e58955ce05227fe108f. Latest consensus state: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } }
I20260926 14:44:45.724545 2637 tablet_replica.cc:442] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:45.725261 2635 tablet_replica.cc:439] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: TxnStatusTablet state changed. Reason: New leader 8a2f6265b49e4e58955ce05227fe108f. Latest consensus state: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } }
I20260926 14:44:45.725364 2635 tablet_replica.cc:442] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:45.727649 2636 tablet_replica.cc:439] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } }
I20260926 14:44:45.727770 2636 tablet_replica.cc:442] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:44:45.729218 2636 tablet_replica.cc:439] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } }
I20260926 14:44:45.729334 2636 tablet_replica.cc:442] T 44846855f260449a9a7eb2bf8f3c9b58 P 8a2f6265b49e4e58955ce05227fe108f: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:44:45.730417 2635 tablet_replica.cc:439] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } }
I20260926 14:44:45.730535 2635 tablet_replica.cc:442] T 44846855f260449a9a7eb2bf8f3c9b58 P 21fe479a9cc3473db1175eec1489afcf: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:45.731583 2637 tablet_replica.cc:439] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } } }
I20260926 14:44:45.731706 2637 tablet_replica.cc:442] T 44846855f260449a9a7eb2bf8f3c9b58 P d32454af21284b218fb244f64b439914: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:44:45.734084 2656 mvcc.cc:205] Tried to move back new op lower bound from 7333617195901755392 to 7333617195809177600. Current Snapshot: MvccSnapshot[applied={T|T < 7333617195901755392}]
I20260926 14:44:45.738381 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.735358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.0.98.1:41933)
0926 14:44:45.735471 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.735476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.735490 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:44:45.735627 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.735630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.735677 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.735748 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:44:45.735983 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.736521 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.737412 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.737661 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.737759 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.737983 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:44:45.738053 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:44:45.738111 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.738205 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.738254 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 14:44:45.739313 2327 tablet.cc:2369] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:44:45.749366 2627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.745926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56651 (local address 127.0.98.1:41933)
0926 14:44:45.746084 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:45.746089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:45.746244 (+ 155us) server_negotiation.cc:407] Connection header received
0926 14:44:45.746372 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:45.746375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:45.746414 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:45.746479 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:44:45.746749 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.747289 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.748217 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.748436 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.748518 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.748877 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:44:45.748973 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:44:45.749048 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:45.749144 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 14:44:45.749202 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":35}
I20260926 14:44:45.749366 2667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:45.745881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41933 (local address 127.0.98.1:56651)
0926 14:44:45.746166 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:45.746178 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:45.746286 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:45.746491 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:45.746493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:45.746500 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:44:45.746660 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.746667 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.747384 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.747387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.748102 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:45.748112 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:45.748555 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:45.748559 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:45.748650 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:45.748694 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:44:45.749067 (+ 373us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:44:45.749073 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:44:45.749118 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 14:44:45.749189 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":17,"thread_start_us":110,"threads_started":1}
I20260926 14:44:45.756481 2560 raft_consensus.cc:1261] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Refusing update from remote peer 8a2f6265b49e4e58955ce05227fe108f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:45.756484 2420 raft_consensus.cc:1261] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Refusing update from remote peer 8a2f6265b49e4e58955ce05227fe108f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:45.756955 2661 consensus_queue.cc:1035] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:45.757144 2640 consensus_queue.cc:1035] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:45.761358 2657 mvcc.cc:205] Tried to move back new op lower bound from 7333617196055347200 to 7333617195423854592. Current Snapshot: MvccSnapshot[applied={T|T < 7333617196055347200}]
I20260926 14:44:45.794646 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:45.794854 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:45.797199 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073: Sending DeleteTablet for 3 replicas of tablet 60b83540a10745299d3e50004045919f
I20260926 14:44:45.797992 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b83540a10745299d3e50004045919f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48226
I20260926 14:44:45.798025 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b83540a10745299d3e50004045919f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41378
I20260926 14:44:45.798025 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 60b83540a10745299d3e50004045919f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37006
I20260926 14:44:45.798727 2672 tablet_replica.cc:330] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914: stopping tablet replica
I20260926 14:44:45.799103 2672 raft_consensus.cc:2229] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:45.799331 2672 raft_consensus.cc:2258] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:45.799330 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:44:45.799711 2673 tablet_replica.cc:330] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf: stopping tablet replica
W20260926 14:44:45.799737 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:44:45.800011 2673 raft_consensus.cc:2229] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:45.800218 2673 raft_consensus.cc:2258] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:45.800704 2671 tablet_replica.cc:330] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: stopping tablet replica
I20260926 14:44:45.800966 2671 raft_consensus.cc:2229] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:44:45.801335 2671 raft_consensus.cc:2258] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:45.801651 2673 ts_tablet_manager.cc:1905] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:45.802687 2671 ts_tablet_manager.cc:1905] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:45.804872 2673 ts_tablet_manager.cc:1918] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:44:45.804978 2673 log.cc:1196] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/60b83540a10745299d3e50004045919f
I20260926 14:44:45.805262 2673 ts_tablet_manager.cc:1939] T 60b83540a10745299d3e50004045919f P 21fe479a9cc3473db1175eec1489afcf: Deleting consensus metadata
I20260926 14:44:45.805491 2671 ts_tablet_manager.cc:1918] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:44:45.805568 2671 log.cc:1196] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/wal/wals/60b83540a10745299d3e50004045919f
I20260926 14:44:45.805836 2671 ts_tablet_manager.cc:1939] T 60b83540a10745299d3e50004045919f P 8a2f6265b49e4e58955ce05227fe108f: Deleting consensus metadata
I20260926 14:44:45.805773 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 1039cb11c86d40e3affd9376c564af46 (DEFAULT_TABLE table=test-table [id=60d64f2fe5d748b289d07e0c16b6a901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:45.806064 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1039cb11c86d40e3affd9376c564af46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:45.806502 2110 catalog_manager.cc:4841] TS 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017): tablet 60b83540a10745299d3e50004045919f (table test-table [id=f39b414afc984a75a5203174c854cc55]) successfully deleted
I20260926 14:44:45.808487 2631 tablet_bootstrap.cc:492] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914: Bootstrap starting.
I20260926 14:44:45.808651 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 1039cb11c86d40e3affd9376c564af46 (DEFAULT_TABLE table=test-table [id=60d64f2fe5d748b289d07e0c16b6a901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:45.808894 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1039cb11c86d40e3affd9376c564af46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:45.809281 2631 tablet_bootstrap.cc:654] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:45.810249 2631 tablet_bootstrap.cc:492] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914: No bootstrap required, opened a new log
I20260926 14:44:45.810312 2631 ts_tablet_manager.cc:1397] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:44:45.810657 2631 raft_consensus.cc:348] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.810747 2631 raft_consensus.cc:374] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:45.810832 2631 raft_consensus.cc:729] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32454af21284b218fb244f64b439914, State: Initialized, Role: FOLLOWER
I20260926 14:44:45.810943 2631 consensus_queue.cc:260] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.811190 2631 ts_tablet_manager.cc:1428] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:44:45.811228 2629 tablet_bootstrap.cc:492] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf: Bootstrap starting.
I20260926 14:44:45.811676 2107 catalog_manager.cc:4841] TS 8a2f6265b49e4e58955ce05227fe108f (127.0.98.1:41933): tablet 60b83540a10745299d3e50004045919f (table test-table [id=f39b414afc984a75a5203174c854cc55]) successfully deleted
I20260926 14:44:45.811851 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 1039cb11c86d40e3affd9376c564af46 (DEFAULT_TABLE table=test-table [id=60d64f2fe5d748b289d07e0c16b6a901]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:45.812038 2629 tablet_bootstrap.cc:654] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:45.812139 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1039cb11c86d40e3affd9376c564af46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:45.814491 2630 tablet_bootstrap.cc:492] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f: Bootstrap starting.
I20260926 14:44:45.815459 2630 tablet_bootstrap.cc:654] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:45.815495 2629 tablet_bootstrap.cc:492] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf: No bootstrap required, opened a new log
I20260926 14:44:45.815575 2629 ts_tablet_manager.cc:1397] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 14:44:45.815629 2672 ts_tablet_manager.cc:1905] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:45.816015 2629 raft_consensus.cc:348] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.816141 2629 raft_consensus.cc:374] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:45.816180 2629 raft_consensus.cc:729] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21fe479a9cc3473db1175eec1489afcf, State: Initialized, Role: FOLLOWER
I20260926 14:44:45.816308 2629 consensus_queue.cc:260] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.816563 2629 ts_tablet_manager.cc:1428] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:44:45.816913 2630 tablet_bootstrap.cc:492] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f: No bootstrap required, opened a new log
I20260926 14:44:45.816989 2630 ts_tablet_manager.cc:1397] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:44:45.817409 2630 raft_consensus.cc:348] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.817528 2630 raft_consensus.cc:374] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:45.817557 2630 raft_consensus.cc:729] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2f6265b49e4e58955ce05227fe108f, State: Initialized, Role: FOLLOWER
I20260926 14:44:45.817674 2630 consensus_queue.cc:260] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.817902 2630 ts_tablet_manager.cc:1428] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:44:45.818907 2672 ts_tablet_manager.cc:1918] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:44:45.819010 2672 log.cc:1196] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/60b83540a10745299d3e50004045919f
I20260926 14:44:45.819295 2672 ts_tablet_manager.cc:1939] T 60b83540a10745299d3e50004045919f P d32454af21284b218fb244f64b439914: Deleting consensus metadata
I20260926 14:44:45.820452 2107 catalog_manager.cc:4841] TS d32454af21284b218fb244f64b439914 (127.0.98.3:44305): tablet 60b83540a10745299d3e50004045919f (table test-table [id=f39b414afc984a75a5203174c854cc55]) successfully deleted
I20260926 14:44:45.905361 2636 raft_consensus.cc:482] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:44:45.905539 2636 raft_consensus.cc:504] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.906001 2636 leader_election.cc:288] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017), d32454af21284b218fb244f64b439914 (127.0.98.3:44305)
I20260926 14:44:45.906508 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1039cb11c86d40e3affd9376c564af46" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fe479a9cc3473db1175eec1489afcf" is_pre_election: true
I20260926 14:44:45.906505 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1039cb11c86d40e3affd9376c564af46" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32454af21284b218fb244f64b439914" is_pre_election: true
I20260926 14:44:45.906692 2419 raft_consensus.cc:2454] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 0.
I20260926 14:44:45.906714 2559 raft_consensus.cc:2454] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 0.
I20260926 14:44:45.907088 2211 leader_election.cc:302] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a2f6265b49e4e58955ce05227fe108f, d32454af21284b218fb244f64b439914; no voters:
I20260926 14:44:45.907302 2636 raft_consensus.cc:2781] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:44:45.907351 2636 raft_consensus.cc:482] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:44:45.907384 2636 raft_consensus.cc:3037] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:45.908610 2636 raft_consensus.cc:504] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.908991 2636 leader_election.cc:288] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 election: Requested vote from peers 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017), d32454af21284b218fb244f64b439914 (127.0.98.3:44305)
I20260926 14:44:45.909360 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1039cb11c86d40e3affd9376c564af46" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fe479a9cc3473db1175eec1489afcf"
I20260926 14:44:45.909502 2419 raft_consensus.cc:3037] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:45.909525 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1039cb11c86d40e3affd9376c564af46" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32454af21284b218fb244f64b439914"
I20260926 14:44:45.909648 2559 raft_consensus.cc:3037] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:45.910526 2419 raft_consensus.cc:2454] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 1.
I20260926 14:44:45.910809 2559 raft_consensus.cc:2454] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 1.
I20260926 14:44:45.910856 2214 leader_election.cc:302] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21fe479a9cc3473db1175eec1489afcf, 8a2f6265b49e4e58955ce05227fe108f; no voters:
I20260926 14:44:45.911017 2636 raft_consensus.cc:2781] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:45.911098 2636 raft_consensus.cc:686] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 1 LEADER]: Becoming Leader. State: Replica: 8a2f6265b49e4e58955ce05227fe108f, State: Running, Role: LEADER
I20260926 14:44:45.911229 2636 consensus_queue.cc:237] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:45.912916 2122 catalog_manager.cc:5495] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2f6265b49e4e58955ce05227fe108f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } health_report { overall_health: UNKNOWN } } }
I20260926 14:44:45.929948 2560 raft_consensus.cc:1261] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Refusing update from remote peer 8a2f6265b49e4e58955ce05227fe108f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:45.930306 2636 consensus_queue.cc:1035] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:45.930392 2419 raft_consensus.cc:1261] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Refusing update from remote peer 8a2f6265b49e4e58955ce05227fe108f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:45.930735 2636 consensus_queue.cc:1035] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:44:45.942714 2655 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 14:44:47.494868 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:47.495039 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:47.497354 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073: Sending DeleteTablet for 3 replicas of tablet 1039cb11c86d40e3affd9376c564af46
I20260926 14:44:47.497856 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 1039cb11c86d40e3affd9376c564af46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48226
I20260926 14:44:47.497925 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 1039cb11c86d40e3affd9376c564af46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37006
I20260926 14:44:47.498021 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 1039cb11c86d40e3affd9376c564af46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41378
I20260926 14:44:47.498191 2702 tablet_replica.cc:330] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f: stopping tablet replica
I20260926 14:44:47.498308 2702 raft_consensus.cc:2229] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:44:47.498487 2702 raft_consensus.cc:2258] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:47.499073 2702 ts_tablet_manager.cc:1905] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:47.501158 2703 tablet_replica.cc:330] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf: stopping tablet replica
I20260926 14:44:47.501284 2703 raft_consensus.cc:2229] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:47.501396 2703 raft_consensus.cc:2258] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:47.501617 2704 tablet_replica.cc:330] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914: stopping tablet replica
I20260926 14:44:47.501751 2704 raft_consensus.cc:2229] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:47.501752 2702 ts_tablet_manager.cc:1918] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:44:47.501861 2704 raft_consensus.cc:2258] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:47.501892 2702 log.cc:1196] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/wal/wals/1039cb11c86d40e3affd9376c564af46
I20260926 14:44:47.501961 2703 ts_tablet_manager.cc:1905] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:47.502288 2702 ts_tablet_manager.cc:1939] T 1039cb11c86d40e3affd9376c564af46 P 8a2f6265b49e4e58955ce05227fe108f: Deleting consensus metadata
I20260926 14:44:47.503037 2107 catalog_manager.cc:4841] TS 8a2f6265b49e4e58955ce05227fe108f (127.0.98.1:41933): tablet 1039cb11c86d40e3affd9376c564af46 (table test-table [id=60d64f2fe5d748b289d07e0c16b6a901]) successfully deleted
I20260926 14:44:47.503726 2704 ts_tablet_manager.cc:1905] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:47.504393 2703 ts_tablet_manager.cc:1918] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:44:47.504467 2703 log.cc:1196] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/1039cb11c86d40e3affd9376c564af46
I20260926 14:44:47.504702 2703 ts_tablet_manager.cc:1939] T 1039cb11c86d40e3affd9376c564af46 P 21fe479a9cc3473db1175eec1489afcf: Deleting consensus metadata
I20260926 14:44:47.505457 2110 catalog_manager.cc:4841] TS 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017): tablet 1039cb11c86d40e3affd9376c564af46 (table test-table [id=60d64f2fe5d748b289d07e0c16b6a901]) successfully deleted
I20260926 14:44:47.505942 2704 ts_tablet_manager.cc:1918] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:44:47.506016 2704 log.cc:1196] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/1039cb11c86d40e3affd9376c564af46
I20260926 14:44:47.506222 2704 ts_tablet_manager.cc:1939] T 1039cb11c86d40e3affd9376c564af46 P d32454af21284b218fb244f64b439914: Deleting consensus metadata
I20260926 14:44:47.506829 2107 catalog_manager.cc:4841] TS d32454af21284b218fb244f64b439914 (127.0.98.3:44305): tablet 1039cb11c86d40e3affd9376c564af46 (table test-table [id=60d64f2fe5d748b289d07e0c16b6a901]) successfully deleted
Sep 26 14:44:47 dist-test-slave-s7k6 krb5kdc[2078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433887, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:44:47 dist-test-slave-s7k6 krb5kdc[2078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433887, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:47.526836 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:47.518346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.0.98.62:46201)
0926 14:44:47.518590 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:47.518596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:47.518609 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:44:47.518649 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:47.518652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:47.518691 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:47.518853 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:47.519634 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:47.520223 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:47.521138 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:47.521412 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:47.521490 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:47.523797 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:47.523800 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:47.523802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:47.523826 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:47.525431 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:47.525779 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:47.525782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:47.525783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:47.525855 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:47.526037 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:47.526039 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:47.526040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:47.526206 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:47.526341 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:47.526508 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 14:44:47.526605 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":88,"threads_started":1}
I20260926 14:44:47.529203 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:44:47.529646 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:44:47.533902 2259 tablet_service.cc:1467] Processing CreateTablet for tablet afe780ffa17640fb9a9d8a569e095cf6 (DEFAULT_TABLE table=test-table [id=f08ebaf6009f4884ba49e02a608fd8d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:47.533912 2400 tablet_service.cc:1467] Processing CreateTablet for tablet afe780ffa17640fb9a9d8a569e095cf6 (DEFAULT_TABLE table=test-table [id=f08ebaf6009f4884ba49e02a608fd8d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:47.534193 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe780ffa17640fb9a9d8a569e095cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:47.534199 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe780ffa17640fb9a9d8a569e095cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:47.534181 2540 tablet_service.cc:1467] Processing CreateTablet for tablet afe780ffa17640fb9a9d8a569e095cf6 (DEFAULT_TABLE table=test-table [id=f08ebaf6009f4884ba49e02a608fd8d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:47.534437 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet afe780ffa17640fb9a9d8a569e095cf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:47.536216 2713 tablet_bootstrap.cc:492] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf: Bootstrap starting.
I20260926 14:44:47.536885 2715 tablet_bootstrap.cc:492] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914: Bootstrap starting.
I20260926 14:44:47.536978 2714 tablet_bootstrap.cc:492] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f: Bootstrap starting.
I20260926 14:44:47.537560 2713 tablet_bootstrap.cc:654] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:47.537725 2715 tablet_bootstrap.cc:654] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:47.537840 2714 tablet_bootstrap.cc:654] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:47.538712 2713 tablet_bootstrap.cc:492] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf: No bootstrap required, opened a new log
I20260926 14:44:47.538828 2713 ts_tablet_manager.cc:1397] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:44:47.538839 2715 tablet_bootstrap.cc:492] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914: No bootstrap required, opened a new log
I20260926 14:44:47.538906 2715 ts_tablet_manager.cc:1397] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:44:47.539288 2713 raft_consensus.cc:348] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.539383 2713 raft_consensus.cc:374] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:47.539413 2713 raft_consensus.cc:729] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21fe479a9cc3473db1175eec1489afcf, State: Initialized, Role: FOLLOWER
I20260926 14:44:47.539357 2715 raft_consensus.cc:348] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.539459 2715 raft_consensus.cc:374] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:47.539498 2715 raft_consensus.cc:729] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32454af21284b218fb244f64b439914, State: Initialized, Role: FOLLOWER
I20260926 14:44:47.539528 2713 consensus_queue.cc:260] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.539611 2715 consensus_queue.cc:260] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.539901 2713 ts_tablet_manager.cc:1428] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:44:47.539966 2715 ts_tablet_manager.cc:1428] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:44:47.540406 2714 tablet_bootstrap.cc:492] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f: No bootstrap required, opened a new log
I20260926 14:44:47.540549 2714 ts_tablet_manager.cc:1397] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 14:44:47.540926 2714 raft_consensus.cc:348] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.541018 2714 raft_consensus.cc:374] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:47.541046 2714 raft_consensus.cc:729] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2f6265b49e4e58955ce05227fe108f, State: Initialized, Role: FOLLOWER
I20260926 14:44:47.541180 2714 consensus_queue.cc:260] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.541430 2714 ts_tablet_manager.cc:1428] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:44:47.542330 2682 raft_consensus.cc:482] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:44:47.542415 2682 raft_consensus.cc:504] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.542783 2682 leader_election.cc:288] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017), d32454af21284b218fb244f64b439914 (127.0.98.3:44305)
I20260926 14:44:47.543130 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe780ffa17640fb9a9d8a569e095cf6" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32454af21284b218fb244f64b439914" is_pre_election: true
I20260926 14:44:47.543130 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe780ffa17640fb9a9d8a569e095cf6" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fe479a9cc3473db1175eec1489afcf" is_pre_election: true
I20260926 14:44:47.543251 2420 raft_consensus.cc:2454] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 0.
I20260926 14:44:47.543272 2559 raft_consensus.cc:2454] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 0.
I20260926 14:44:47.543661 2211 leader_election.cc:302] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a2f6265b49e4e58955ce05227fe108f, d32454af21284b218fb244f64b439914; no voters:
I20260926 14:44:47.543879 2682 raft_consensus.cc:2781] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:44:47.543929 2682 raft_consensus.cc:482] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:44:47.543954 2682 raft_consensus.cc:3037] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:47.544869 2682 raft_consensus.cc:504] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.545159 2682 leader_election.cc:288] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 election: Requested vote from peers 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017), d32454af21284b218fb244f64b439914 (127.0.98.3:44305)
I20260926 14:44:47.545480 2559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe780ffa17640fb9a9d8a569e095cf6" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32454af21284b218fb244f64b439914"
I20260926 14:44:47.545606 2559 raft_consensus.cc:3037] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:47.546546 2559 raft_consensus.cc:2454] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 1.
I20260926 14:44:47.546782 2420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "afe780ffa17640fb9a9d8a569e095cf6" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fe479a9cc3473db1175eec1489afcf"
I20260926 14:44:47.546797 2211 leader_election.cc:302] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a2f6265b49e4e58955ce05227fe108f, d32454af21284b218fb244f64b439914; no voters:
I20260926 14:44:47.546898 2420 raft_consensus.cc:3037] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:47.546963 2682 raft_consensus.cc:2781] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:47.547037 2682 raft_consensus.cc:686] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 1 LEADER]: Becoming Leader. State: Replica: 8a2f6265b49e4e58955ce05227fe108f, State: Running, Role: LEADER
I20260926 14:44:47.547165 2682 consensus_queue.cc:237] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.547972 2420 raft_consensus.cc:2454] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 1.
I20260926 14:44:47.548765 2121 catalog_manager.cc:5495] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2f6265b49e4e58955ce05227fe108f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } health_report { overall_health: UNKNOWN } } }
I20260926 14:44:47.566020 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:47.563177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.0.98.1:41933)
0926 14:44:47.563420 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:47.563427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:47.563438 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:44:47.563474 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:47.563476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:47.563515 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:47.563583 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:44:47.563811 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:47.564361 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:47.565127 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:47.565372 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:47.565447 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:47.565649 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:44:47.565717 (+ 68us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:44:47.565770 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:47.565825 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 14:44:47.565866 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":104,"threads_started":1}
I20260926 14:44:47.571436 2559 raft_consensus.cc:1261] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Refusing update from remote peer 8a2f6265b49e4e58955ce05227fe108f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:47.571534 2420 raft_consensus.cc:1261] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Refusing update from remote peer 8a2f6265b49e4e58955ce05227fe108f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:47.571767 2682 consensus_queue.cc:1035] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:47.571920 2690 consensus_queue.cc:1035] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:47.574079 2683 mvcc.cc:205] Tried to move back new op lower bound from 7333617203490045952 to 7333617203394449408. Current Snapshot: MvccSnapshot[applied={T|T < 7333617203490045952}]
I20260926 14:44:47.596882 2122 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:47.597029 2122 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:47.598819 2122 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073: Sending DeleteTablet for 3 replicas of tablet afe780ffa17640fb9a9d8a569e095cf6
I20260926 14:44:47.599284 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet afe780ffa17640fb9a9d8a569e095cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48226
I20260926 14:44:47.599397 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet afe780ffa17640fb9a9d8a569e095cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41378
I20260926 14:44:47.599597 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet afe780ffa17640fb9a9d8a569e095cf6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37006
I20260926 14:44:47.599787 2702 tablet_replica.cc:330] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f: stopping tablet replica
I20260926 14:44:47.599781 2703 tablet_replica.cc:330] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf: stopping tablet replica
I20260926 14:44:47.599908 2702 raft_consensus.cc:2229] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:44:47.599907 2703 raft_consensus.cc:2229] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:47.599943 2704 tablet_replica.cc:330] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914: stopping tablet replica
I20260926 14:44:47.600013 2703 raft_consensus.cc:2258] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:47.600046 2702 raft_consensus.cc:2258] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:47.600044 2704 raft_consensus.cc:2229] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:47.600137 2704 raft_consensus.cc:2258] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:47.600309 2122 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:44:47.600698 2122 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:44:47.600741 2704 ts_tablet_manager.cc:1905] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:47.600937 2703 ts_tablet_manager.cc:1905] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:47.601800 2702 ts_tablet_manager.cc:1905] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:47.603104 2704 ts_tablet_manager.cc:1918] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:44:47.603163 2704 log.cc:1196] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/afe780ffa17640fb9a9d8a569e095cf6
I20260926 14:44:47.603323 2704 ts_tablet_manager.cc:1939] T afe780ffa17640fb9a9d8a569e095cf6 P d32454af21284b218fb244f64b439914: Deleting consensus metadata
I20260926 14:44:47.603595 2703 ts_tablet_manager.cc:1918] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:44:47.603658 2703 log.cc:1196] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/afe780ffa17640fb9a9d8a569e095cf6
I20260926 14:44:47.603889 2703 ts_tablet_manager.cc:1939] T afe780ffa17640fb9a9d8a569e095cf6 P 21fe479a9cc3473db1175eec1489afcf: Deleting consensus metadata
I20260926 14:44:47.603888 2107 catalog_manager.cc:4841] TS d32454af21284b218fb244f64b439914 (127.0.98.3:44305): tablet afe780ffa17640fb9a9d8a569e095cf6 (table test-table [id=f08ebaf6009f4884ba49e02a608fd8d1]) successfully deleted
I20260926 14:44:47.604192 2702 ts_tablet_manager.cc:1918] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:44:47.604251 2702 log.cc:1196] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-0/wal/wals/afe780ffa17640fb9a9d8a569e095cf6
I20260926 14:44:47.604473 2702 ts_tablet_manager.cc:1939] T afe780ffa17640fb9a9d8a569e095cf6 P 8a2f6265b49e4e58955ce05227fe108f: Deleting consensus metadata
I20260926 14:44:47.604449 2110 catalog_manager.cc:4841] TS 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017): tablet afe780ffa17640fb9a9d8a569e095cf6 (table test-table [id=f08ebaf6009f4884ba49e02a608fd8d1]) successfully deleted
I20260926 14:44:47.604996 2107 catalog_manager.cc:4841] TS 8a2f6265b49e4e58955ce05227fe108f (127.0.98.1:41933): tablet afe780ffa17640fb9a9d8a569e095cf6 (table test-table [id=f08ebaf6009f4884ba49e02a608fd8d1]) successfully deleted
I20260926 14:44:47.607287 2259 tablet_service.cc:1467] Processing CreateTablet for tablet 78d521b0daa74430b806b75c305ab60a (DEFAULT_TABLE table=test-table [id=73ad69038b56482c8d07437cdea7609e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:47.607363 2400 tablet_service.cc:1467] Processing CreateTablet for tablet 78d521b0daa74430b806b75c305ab60a (DEFAULT_TABLE table=test-table [id=73ad69038b56482c8d07437cdea7609e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:47.607537 2259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d521b0daa74430b806b75c305ab60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:47.607509 2540 tablet_service.cc:1467] Processing CreateTablet for tablet 78d521b0daa74430b806b75c305ab60a (DEFAULT_TABLE table=test-table [id=73ad69038b56482c8d07437cdea7609e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:47.607596 2400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d521b0daa74430b806b75c305ab60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:47.607704 2540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78d521b0daa74430b806b75c305ab60a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:47.609370 2714 tablet_bootstrap.cc:492] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f: Bootstrap starting.
I20260926 14:44:47.609618 2715 tablet_bootstrap.cc:492] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914: Bootstrap starting.
I20260926 14:44:47.609673 2713 tablet_bootstrap.cc:492] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf: Bootstrap starting.
I20260926 14:44:47.610174 2714 tablet_bootstrap.cc:654] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:47.610360 2713 tablet_bootstrap.cc:654] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:47.610447 2715 tablet_bootstrap.cc:654] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:47.611263 2714 tablet_bootstrap.cc:492] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f: No bootstrap required, opened a new log
I20260926 14:44:47.611308 2713 tablet_bootstrap.cc:492] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf: No bootstrap required, opened a new log
I20260926 14:44:47.611351 2714 ts_tablet_manager.cc:1397] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:44:47.611382 2713 ts_tablet_manager.cc:1397] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:44:47.611687 2715 tablet_bootstrap.cc:492] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914: No bootstrap required, opened a new log
I20260926 14:44:47.611743 2715 ts_tablet_manager.cc:1397] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:44:47.611712 2714 raft_consensus.cc:348] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.611739 2713 raft_consensus.cc:348] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.611821 2713 raft_consensus.cc:374] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:47.611816 2714 raft_consensus.cc:374] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:47.611850 2713 raft_consensus.cc:729] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21fe479a9cc3473db1175eec1489afcf, State: Initialized, Role: FOLLOWER
I20260926 14:44:47.611860 2714 raft_consensus.cc:729] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a2f6265b49e4e58955ce05227fe108f, State: Initialized, Role: FOLLOWER
I20260926 14:44:47.611963 2713 consensus_queue.cc:260] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.611963 2714 consensus_queue.cc:260] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.612077 2715 raft_consensus.cc:348] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.612175 2715 raft_consensus.cc:374] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:47.612203 2715 raft_consensus.cc:729] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d32454af21284b218fb244f64b439914, State: Initialized, Role: FOLLOWER
I20260926 14:44:47.612206 2714 ts_tablet_manager.cc:1428] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:44:47.612206 2713 ts_tablet_manager.cc:1428] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:44:47.612305 2715 consensus_queue.cc:260] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.612543 2715 ts_tablet_manager.cc:1428] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:44:47.672654 2685 raft_consensus.cc:482] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:44:47.672788 2685 raft_consensus.cc:504] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.673224 2685 leader_election.cc:288] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017), d32454af21284b218fb244f64b439914 (127.0.98.3:44305)
I20260926 14:44:47.673671 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d521b0daa74430b806b75c305ab60a" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fe479a9cc3473db1175eec1489afcf" is_pre_election: true
I20260926 14:44:47.673648 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d521b0daa74430b806b75c305ab60a" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32454af21284b218fb244f64b439914" is_pre_election: true
I20260926 14:44:47.673805 2560 raft_consensus.cc:2454] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 0.
I20260926 14:44:47.673834 2419 raft_consensus.cc:2454] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 0.
I20260926 14:44:47.674115 2211 leader_election.cc:302] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a2f6265b49e4e58955ce05227fe108f, d32454af21284b218fb244f64b439914; no voters:
I20260926 14:44:47.674285 2685 raft_consensus.cc:2781] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:44:47.674379 2685 raft_consensus.cc:482] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:44:47.674413 2685 raft_consensus.cc:3037] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:47.675455 2685 raft_consensus.cc:504] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.675773 2685 leader_election.cc:288] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 election: Requested vote from peers 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017), d32454af21284b218fb244f64b439914 (127.0.98.3:44305)
I20260926 14:44:47.676126 2419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d521b0daa74430b806b75c305ab60a" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21fe479a9cc3473db1175eec1489afcf"
I20260926 14:44:47.676173 2560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78d521b0daa74430b806b75c305ab60a" candidate_uuid: "8a2f6265b49e4e58955ce05227fe108f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d32454af21284b218fb244f64b439914"
I20260926 14:44:47.676254 2419 raft_consensus.cc:3037] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:47.676271 2560 raft_consensus.cc:3037] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:47.677313 2560 raft_consensus.cc:2454] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 1.
I20260926 14:44:47.677327 2419 raft_consensus.cc:2454] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a2f6265b49e4e58955ce05227fe108f in term 1.
I20260926 14:44:47.677592 2214 leader_election.cc:302] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21fe479a9cc3473db1175eec1489afcf, 8a2f6265b49e4e58955ce05227fe108f; no voters:
I20260926 14:44:47.677759 2685 raft_consensus.cc:2781] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:47.677888 2685 raft_consensus.cc:686] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [term 1 LEADER]: Becoming Leader. State: Replica: 8a2f6265b49e4e58955ce05227fe108f, State: Running, Role: LEADER
I20260926 14:44:47.678022 2685 consensus_queue.cc:237] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } }
I20260926 14:44:47.679692 2121 catalog_manager.cc:5495] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a2f6265b49e4e58955ce05227fe108f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8a2f6265b49e4e58955ce05227fe108f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a2f6265b49e4e58955ce05227fe108f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 } health_report { overall_health: UNKNOWN } } }
I20260926 14:44:47.699218 2560 raft_consensus.cc:1261] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Refusing update from remote peer 8a2f6265b49e4e58955ce05227fe108f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:47.699257 2419 raft_consensus.cc:1261] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Refusing update from remote peer 8a2f6265b49e4e58955ce05227fe108f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:47.699594 2685 consensus_queue.cc:1035] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Connected to new peer: Peer: permanent_uuid: "21fe479a9cc3473db1175eec1489afcf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:47.699774 2682 consensus_queue.cc:1035] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d32454af21284b218fb244f64b439914" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:44:47.712134 2683 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 14:44:49.257367 2121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:49.257524 2121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:49.259789 2121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d83951ff50414297ba59687f4c76d073: Sending DeleteTablet for 3 replicas of tablet 78d521b0daa74430b806b75c305ab60a
I20260926 14:44:49.260284 2259 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d521b0daa74430b806b75c305ab60a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48226
I20260926 14:44:49.260321 2400 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d521b0daa74430b806b75c305ab60a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37006
I20260926 14:44:49.260336 2540 tablet_service.cc:1514] Processing DeleteTablet for tablet 78d521b0daa74430b806b75c305ab60a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41378
I20260926 14:44:49.260634 2753 tablet_replica.cc:330] T 78d521b0daa74430b806b75c305ab60a P 8a2f6265b49e4e58955ce05227fe108f: stopping tablet replica
I20260926 14:44:49.260653 2755 tablet_replica.cc:330] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914: stopping tablet replica
I20260926 14:44:49.260737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 2195
I20260926 14:44:49.260771 2755 raft_consensus.cc:2229] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:49.261521 2754 tablet_replica.cc:330] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf: stopping tablet replica
I20260926 14:44:49.261629 2754 raft_consensus.cc:2229] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:49.261724 2754 raft_consensus.cc:2258] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:49.262349 2754 ts_tablet_manager.cc:1905] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:49.262542 2755 raft_consensus.cc:2258] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:49.263084 2755 ts_tablet_manager.cc:1905] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:49.264974 2754 ts_tablet_manager.cc:1918] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:44:49.264997 2755 ts_tablet_manager.cc:1918] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 14:44:49.265051 2755 log.cc:1196] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/78d521b0daa74430b806b75c305ab60a
I20260926 14:44:49.265054 2754 log.cc:1196] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/78d521b0daa74430b806b75c305ab60a
I20260926 14:44:49.265278 2755 ts_tablet_manager.cc:1939] T 78d521b0daa74430b806b75c305ab60a P d32454af21284b218fb244f64b439914: Deleting consensus metadata
I20260926 14:44:49.265314 2754 ts_tablet_manager.cc:1939] T 78d521b0daa74430b806b75c305ab60a P 21fe479a9cc3473db1175eec1489afcf: Deleting consensus metadata
I20260926 14:44:49.266000 2107 catalog_manager.cc:4841] TS d32454af21284b218fb244f64b439914 (127.0.98.3:44305): tablet 78d521b0daa74430b806b75c305ab60a (table test-table [id=73ad69038b56482c8d07437cdea7609e]) successfully deleted
I20260926 14:44:49.266175 2110 catalog_manager.cc:4841] TS 21fe479a9cc3473db1175eec1489afcf (127.0.98.2:38017): tablet 78d521b0daa74430b806b75c305ab60a (table test-table [id=73ad69038b56482c8d07437cdea7609e]) successfully deleted
W20260926 14:44:49.268286 2107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41933 (error 108)
I20260926 14:44:49.268994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 2335
I20260926 14:44:49.269881 2757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:49.269001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41933 (local address 127.0.0.1:48266)
0926 14:44:49.269702 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:49.269776 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":71,"threads_started":1}
W20260926 14:44:49.270290 2107 catalog_manager.cc:4584] TS 8a2f6265b49e4e58955ce05227fe108f (127.0.98.1:41933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 78d521b0daa74430b806b75c305ab60a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41933: connect: Connection refused (error 111)
I20260926 14:44:49.274535 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 2475
I20260926 14:44:49.279423 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 2091
2026-09-26T14:44:49Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5864 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:44:49 dist-test-slave-s7k6 krb5kdc[2760](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:44:49 dist-test-slave-s7k6 krb5kdc[2760](info): set up 2 sockets
Sep 26 14:44:49 dist-test-slave-s7k6 krb5kdc[2760](info): commencing operation
krb5kdc: starting...
W20260926 14:44:50.242486 392 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:44:50 dist-test-slave-s7k6 krb5kdc[2760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:44:50.369306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:50.478101 2773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:50.478384 2773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:50.478423 2773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:50.482474 2773 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:44:50.482543 2773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:50.482561 2773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:50.482575 2773 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:44:50.482589 2773 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:44:50.487866 2773 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:50.488866 2773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:50.489838 2773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:50.497898 2779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:50.497932 2778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:50.497901 2781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:50.498191 2773 server_base.cc:1034] running on GCE node
I20260926 14:44:50.498826 2773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:50.499967 2773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:50.501147 2773 hybrid_clock.cc:648] HybridClock initialized: now 1790433890501126 us; error 33 us; skew 500 ppm
Sep 26 14:44:50 dist-test-slave-s7k6 krb5kdc[2760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:50.508515 2773 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:44:50.509609 2773 webserver.cc:460] Webserver started at http://127.0.98.62:43097/ using document root <none> and password file <none>
I20260926 14:44:50.510212 2773 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:50.510278 2773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:50.510479 2773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:50.512657 2773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "4964dfe7c45b4a0ebeb3e1602cd3a1e9"
format_stamp: "Formatted at 2026-09-26 14:44:50 on dist-test-slave-s7k6"
server_key: "7830715111f070669f022003fd6629f8"
server_key_iv: "662038bdbd8c6675ba5322db4236c2fa"
server_key_version: "encryptionkey@0"
I20260926 14:44:50.513242 2773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "4964dfe7c45b4a0ebeb3e1602cd3a1e9"
format_stamp: "Formatted at 2026-09-26 14:44:50 on dist-test-slave-s7k6"
server_key: "7830715111f070669f022003fd6629f8"
server_key_iv: "662038bdbd8c6675ba5322db4236c2fa"
server_key_version: "encryptionkey@0"
I20260926 14:44:50.517336 2773 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:44:50.519862 2788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:50.521201 2773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:50.521333 2773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "4964dfe7c45b4a0ebeb3e1602cd3a1e9"
format_stamp: "Formatted at 2026-09-26 14:44:50 on dist-test-slave-s7k6"
server_key: "7830715111f070669f022003fd6629f8"
server_key_iv: "662038bdbd8c6675ba5322db4236c2fa"
server_key_version: "encryptionkey@0"
I20260926 14:44:50.521456 2773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:50.533612 2773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:50.536542 2773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:50.536712 2773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:50.543311 2773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45019
I20260926 14:44:50.543336 2840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45019 every 8 connection(s)
I20260926 14:44:50.544523 2773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:44:50.545146 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 2773
I20260926 14:44:50.545234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:44:50.545485 392 external_mini_cluster.cc:1428] Setting key 521a5b7b3bda5a4cb5280a29d74c03d2
I20260926 14:44:50.547742 2841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:50.554275 2841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9: Bootstrap starting.
Sep 26 14:44:50 dist-test-slave-s7k6 krb5kdc[2760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433890, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:50.556864 2841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:50.557709 2841 log.cc:824] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:50.560081 2841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9: No bootstrap required, opened a new log
I20260926 14:44:50.560171 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:50.546886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.0.98.62:45019)
0926 14:44:50.547287 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:50.547309 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:50.547353 (+ 44us) server_negotiation.cc:407] Connection header received
0926 14:44:50.548196 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:50.548221 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:50.548652 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:50.549034 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:50.549703 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:50.551206 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:50.552093 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:50.552764 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:50.552930 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:50.555328 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:50.555335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:50.555352 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:50.555385 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:50.557205 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:50.557617 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:50.557623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:50.557629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:50.557772 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:50.558045 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:50.558048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:50.558050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:50.558476 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:50.558665 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:50.558926 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 14:44:50.559224 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":105,"threads_started":1}
I20260926 14:44:50.563298 2841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4964dfe7c45b4a0ebeb3e1602cd3a1e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260926 14:44:50.563510 2841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:50.563544 2841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4964dfe7c45b4a0ebeb3e1602cd3a1e9, State: Initialized, Role: FOLLOWER
I20260926 14:44:50.563889 2841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4964dfe7c45b4a0ebeb3e1602cd3a1e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260926 14:44:50.563997 2841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:44:50.564033 2841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:44:50.564100 2841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:50.564952 2841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4964dfe7c45b4a0ebeb3e1602cd3a1e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260926 14:44:50.565256 2841 leader_election.cc:302] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4964dfe7c45b4a0ebeb3e1602cd3a1e9; no voters:
I20260926 14:44:50.565498 2841 leader_election.cc:288] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:44:50.565613 2846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:50.565815 2846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [term 1 LEADER]: Becoming Leader. State: Replica: 4964dfe7c45b4a0ebeb3e1602cd3a1e9, State: Running, Role: LEADER
I20260926 14:44:50.566164 2846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4964dfe7c45b4a0ebeb3e1602cd3a1e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } }
I20260926 14:44:50.566639 2841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:44:50.567763 2848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4964dfe7c45b4a0ebeb3e1602cd3a1e9. Latest consensus state: current_term: 1 leader_uuid: "4964dfe7c45b4a0ebeb3e1602cd3a1e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4964dfe7c45b4a0ebeb3e1602cd3a1e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } }
I20260926 14:44:50.567885 2848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:50.568141 2847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4964dfe7c45b4a0ebeb3e1602cd3a1e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4964dfe7c45b4a0ebeb3e1602cd3a1e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45019 } } }
I20260926 14:44:50.568255 2847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9 [sys.catalog]: This master's current role is: LEADER
W20260926 14:44:50.571036 2861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:44:50.571110 2861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:44:50.571216 2855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:44:50.572109 2855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:44:50.577610 2855 catalog_manager.cc:1347] Generated new cluster ID: 042913152e5b45718e90f6295e7d933a
I20260926 14:44:50.577688 2855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:44:50.591625 2855 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:44:50.592916 2855 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:44:50.600894 2855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4964dfe7c45b4a0ebeb3e1602cd3a1e9: Generated new TSK 0
I20260926 14:44:50.601719 2855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:44:50.640029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:50.750859 2867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:50.751148 2867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:50.751194 2867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:50.755311 2867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:50.755371 2867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:50.755434 2867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:44:50.760753 2867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:50.761776 2867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:50.762812 2867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:44:50.772249 2867 server_base.cc:1034] running on GCE node
W20260926 14:44:50.772178 2873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:50.772189 2872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:50.772198 2875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:50.772847 2867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:50.773481 2867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:50.774715 2867 hybrid_clock.cc:648] HybridClock initialized: now 1790433890774697 us; error 33 us; skew 500 ppm
Sep 26 14:44:50 dist-test-slave-s7k6 krb5kdc[2760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:50.777667 2867 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:44:50.778544 2867 webserver.cc:460] Webserver started at http://127.0.98.1:44075/ using document root <none> and password file <none>
I20260926 14:44:50.779124 2867 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:50.779187 2867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:50.779350 2867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:50.781409 2867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "3a652bd8a3da4328b08c0cb439d5fac8"
format_stamp: "Formatted at 2026-09-26 14:44:50 on dist-test-slave-s7k6"
server_key: "bf65f8d328649aa73d3a02c3bb3e5229"
server_key_iv: "2e562ff2c5994eac3dfd2308112b02df"
server_key_version: "encryptionkey@0"
I20260926 14:44:50.781932 2867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3a652bd8a3da4328b08c0cb439d5fac8"
format_stamp: "Formatted at 2026-09-26 14:44:50 on dist-test-slave-s7k6"
server_key: "bf65f8d328649aa73d3a02c3bb3e5229"
server_key_iv: "2e562ff2c5994eac3dfd2308112b02df"
server_key_version: "encryptionkey@0"
I20260926 14:44:50.785531 2867 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:44:50.788100 2882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:50.789477 2867 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:44:50.789583 2867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "3a652bd8a3da4328b08c0cb439d5fac8"
format_stamp: "Formatted at 2026-09-26 14:44:50 on dist-test-slave-s7k6"
server_key: "bf65f8d328649aa73d3a02c3bb3e5229"
server_key_iv: "2e562ff2c5994eac3dfd2308112b02df"
server_key_version: "encryptionkey@0"
I20260926 14:44:50.789698 2867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:50.811918 2867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:50.815260 2867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:50.815448 2867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:50.816093 2867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:50.817094 2867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:50.817149 2867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:50.817204 2867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:50.817233 2867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:50.825395 2867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39803
I20260926 14:44:50.825465 2995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39803 every 8 connection(s)
I20260926 14:44:50.826292 2867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:44:50.827291 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 2867
I20260926 14:44:50.827384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:44:50.827636 392 external_mini_cluster.cc:1428] Setting key 954fd2f9024eb08d171028e991147803
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Sep 26 14:44:50 dist-test-slave-s7k6 krb5kdc[2760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 14:44:50.839906 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:50.828163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53091 (local address 127.0.98.62:45019)
0926 14:44:50.828326 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:50.828330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:50.829298 (+ 968us) server_negotiation.cc:407] Connection header received
0926 14:44:50.830826 (+ 1528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:50.830830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:50.830887 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:50.830959 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:50.832158 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:50.832681 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:50.833813 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:50.834041 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:50.834132 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:50.836577 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:50.836580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:50.836582 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:50.836608 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:50.837893 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:50.838410 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:50.838413 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:50.838414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:50.838487 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:50.838822 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:50.838826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:50.838828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:50.839011 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:50.839098 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:50.839608 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 14:44:50.839740 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 14:44:50.840745 2998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:50.828649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.98.1:53091)
0926 14:44:50.829152 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:50.829208 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:50.830584 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:50.831130 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:50.831138 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:50.831494 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:50.832002 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:50.832016 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:50.832800 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:50.832803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:50.833696 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:50.833705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:50.834187 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:50.834191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:50.834356 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:50.834953 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:50.834970 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:50.836404 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:50.838004 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:50.838010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:50.838026 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:50.838314 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:50.838568 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:50.838571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:50.838573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:50.838721 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:50.839108 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:50.839115 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:50.839372 (+ 257us) client_negotiation.cc:769] Sending connection context
0926 14:44:50.839558 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 14:44:50.839908 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":153,"threads_started":1}
I20260926 14:44:50.841925 2996 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45019
I20260926 14:44:50.842157 2996 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:50.842633 2996 heartbeater.cc:511] Master 127.0.98.62:45019 requested a full tablet report, sending...
I20260926 14:44:50.844316 2805 ts_manager.cc:194] Registered new tserver with Master: 3a652bd8a3da4328b08c0cb439d5fac8 (127.0.98.1:39803)
I20260926 14:44:50.845660 2805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:44:50.851975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:50.960193 3001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:50.960464 3001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:50.960506 3001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:50.964789 3001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:50.964848 3001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:50.964907 3001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:44:50.970235 3001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:50.971262 3001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:50.972208 3001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:50.981570 3009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:50.981554 3006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:50.981554 3007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:50.981743 3001 server_base.cc:1034] running on GCE node
I20260926 14:44:50.982198 3001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:50.982875 3001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:50.984045 3001 hybrid_clock.cc:648] HybridClock initialized: now 1790433890984024 us; error 57 us; skew 500 ppm
Sep 26 14:44:50 dist-test-slave-s7k6 krb5kdc[2760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:50.986771 3001 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:44:50.987691 3001 webserver.cc:460] Webserver started at http://127.0.98.2:45577/ using document root <none> and password file <none>
I20260926 14:44:50.988233 3001 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:50.988296 3001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:50.988484 3001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:50.990554 3001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "beb7f368f89a4bb391fd23e3f45eb585"
format_stamp: "Formatted at 2026-09-26 14:44:50 on dist-test-slave-s7k6"
server_key: "6bc0439642091b9a91d5e7cc61c6d1ed"
server_key_iv: "a28f18f919a3571ed2bddd24c4ea0420"
server_key_version: "encryptionkey@0"
I20260926 14:44:50.991146 3001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "beb7f368f89a4bb391fd23e3f45eb585"
format_stamp: "Formatted at 2026-09-26 14:44:50 on dist-test-slave-s7k6"
server_key: "6bc0439642091b9a91d5e7cc61c6d1ed"
server_key_iv: "a28f18f919a3571ed2bddd24c4ea0420"
server_key_version: "encryptionkey@0"
I20260926 14:44:50.995021 3001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:44:50.997543 3016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:50.998888 3001 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:50.998987 3001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "beb7f368f89a4bb391fd23e3f45eb585"
format_stamp: "Formatted at 2026-09-26 14:44:50 on dist-test-slave-s7k6"
server_key: "6bc0439642091b9a91d5e7cc61c6d1ed"
server_key_iv: "a28f18f919a3571ed2bddd24c4ea0420"
server_key_version: "encryptionkey@0"
I20260926 14:44:50.999096 3001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:51.051586 3001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:51.054744 3001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:51.054937 3001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:51.055521 3001 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:51.056337 3001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:51.056383 3001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:51.056422 3001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:51.056437 3001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:51.064674 3001 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40257
I20260926 14:44:51.064819 3129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40257 every 8 connection(s)
I20260926 14:44:51.065577 3001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:44:51.068634 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 3001
I20260926 14:44:51.068754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:44:51.069048 392 external_mini_cluster.cc:1428] Setting key 41ea69bc682331b0bbffcde64becfbc7
Sep 26 14:44:51 dist-test-slave-s7k6 krb5kdc[2760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:44:51.081295 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:51.067424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42043 (local address 127.0.98.62:45019)
0926 14:44:51.067546 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:51.067550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:51.068593 (+ 1043us) server_negotiation.cc:407] Connection header received
0926 14:44:51.070309 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:51.070313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:51.070367 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:51.070441 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:51.071759 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:51.072351 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:51.073994 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:51.074237 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:51.074355 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:51.077252 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:51.077256 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:51.077259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:51.077289 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:51.078953 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:51.079578 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:51.079581 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:51.079582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:51.079716 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:51.080095 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:51.080097 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:51.080099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:51.080294 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:51.080370 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:51.081043 (+ 673us) server_negotiation.cc:299] Negotiation successful
0926 14:44:51.081165 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:44:51.082453 3132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:51.067871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.98.2:42043)
0926 14:44:51.068410 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:51.068472 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:51.070042 (+ 1570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:51.070647 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:51.070655 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:51.071067 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:51.071572 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:51.071587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:51.072474 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:51.072478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:51.073850 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:51.073861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:51.074417 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:51.074421 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:51.074636 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:51.075305 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:51.075327 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:51.077033 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:51.079072 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:51.079077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:51.079092 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:51.079439 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:51.079844 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:51.079846 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:51.079848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:51.079998 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:51.080394 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:51.080400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:51.080765 (+ 365us) client_negotiation.cc:769] Sending connection context
0926 14:44:51.080972 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 14:44:51.081324 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260926 14:44:51.083711 3130 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45019
I20260926 14:44:51.083976 3130 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:51.084535 3130 heartbeater.cc:511] Master 127.0.98.62:45019 requested a full tablet report, sending...
I20260926 14:44:51.085742 2805 ts_manager.cc:194] Registered new tserver with Master: beb7f368f89a4bb391fd23e3f45eb585 (127.0.98.2:40257)
I20260926 14:44:51.086328 2805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42043
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:44:51.095557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:51.205169 3135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:51.205492 3135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:51.205566 3135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:51.209734 3135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:51.209798 3135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:51.209869 3135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:44:51.215229 3135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45019
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:51.216286 3135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:51.217306 3135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:51.226734 3140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:51.226738 3143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:51.226738 3141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:51.227152 3135 server_base.cc:1034] running on GCE node
I20260926 14:44:51.227530 3135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:51.228137 3135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:51.229306 3135 hybrid_clock.cc:648] HybridClock initialized: now 1790433891229284 us; error 34 us; skew 500 ppm
Sep 26 14:44:51 dist-test-slave-s7k6 krb5kdc[2760](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:51.232199 3135 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:44:51.233130 3135 webserver.cc:460] Webserver started at http://127.0.98.3:34715/ using document root <none> and password file <none>
I20260926 14:44:51.233680 3135 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:51.233747 3135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:51.233942 3135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:51.236028 3135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "ffcf4f0a76ce4cda90d9a440b0d36ee9"
format_stamp: "Formatted at 2026-09-26 14:44:51 on dist-test-slave-s7k6"
server_key: "9a1388dbe980de15c00f4cbf9b7ce7d6"
server_key_iv: "a8bd75f652d0d9ef4996a9476b68c900"
server_key_version: "encryptionkey@0"
I20260926 14:44:51.236557 3135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ffcf4f0a76ce4cda90d9a440b0d36ee9"
format_stamp: "Formatted at 2026-09-26 14:44:51 on dist-test-slave-s7k6"
server_key: "9a1388dbe980de15c00f4cbf9b7ce7d6"
server_key_iv: "a8bd75f652d0d9ef4996a9476b68c900"
server_key_version: "encryptionkey@0"
I20260926 14:44:51.240259 3135 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:44:51.242662 3150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:51.243929 3135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:51.244040 3135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "ffcf4f0a76ce4cda90d9a440b0d36ee9"
format_stamp: "Formatted at 2026-09-26 14:44:51 on dist-test-slave-s7k6"
server_key: "9a1388dbe980de15c00f4cbf9b7ce7d6"
server_key_iv: "a8bd75f652d0d9ef4996a9476b68c900"
server_key_version: "encryptionkey@0"
I20260926 14:44:51.244133 3135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:51.259326 3135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:51.262183 3135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:51.262348 3135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:51.262945 3135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:51.263788 3135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:51.263836 3135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:51.263887 3135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:51.263912 3135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:51.272523 3135 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42663
I20260926 14:44:51.272540 3263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42663 every 8 connection(s)
I20260926 14:44:51.273429 3135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:44:51.281456 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 3135
I20260926 14:44:51.281566 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:44:51.281862 392 external_mini_cluster.cc:1428] Setting key b039a2f1c3aaf43fea256695b156cdfc
Sep 26 14:44:51 dist-test-slave-s7k6 krb5kdc[2760](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:51.287128 2844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:51.275210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40295 (local address 127.0.98.62:45019)
0926 14:44:51.275340 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:51.275344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:51.276221 (+ 877us) server_negotiation.cc:407] Connection header received
0926 14:44:51.277573 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:51.277585 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:51.277644 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:51.277762 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:51.278963 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:51.279764 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:51.280903 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:51.281132 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:51.281220 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:51.284057 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:51.284061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:51.284063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:51.284089 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:51.285204 (+ 1115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:51.285662 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:51.285666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:51.285668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:51.285754 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:51.286063 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:51.286066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:51.286067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:51.286248 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:51.286334 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:51.286846 (+ 512us) server_negotiation.cc:299] Negotiation successful
0926 14:44:51.286967 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:44:51.287966 3266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:51.275605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45019 (local address 127.0.98.3:40295)
0926 14:44:51.276063 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:51.276107 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:51.277367 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:51.277928 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:51.277934 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:51.278299 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:51.278831 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:51.278847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:51.279890 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:51.279893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:51.280783 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:51.280792 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:51.281296 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:51.281300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:51.281455 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:51.282112 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:51.282128 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:51.283901 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:51.285307 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:51.285312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:51.285329 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:51.285570 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:51.285836 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:51.285839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:51.285841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:51.285977 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:51.286343 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:51.286349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:51.286598 (+ 249us) client_negotiation.cc:769] Sending connection context
0926 14:44:51.286791 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 14:44:51.287128 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260926 14:44:51.289012 3264 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45019
I20260926 14:44:51.289258 3264 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:51.289728 3264 heartbeater.cc:511] Master 127.0.98.62:45019 requested a full tablet report, sending...
I20260926 14:44:51.290680 2805 ts_manager.cc:194] Registered new tserver with Master: ffcf4f0a76ce4cda90d9a440b0d36ee9 (127.0.98.3:42663)
I20260926 14:44:51.291275 2805 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40295
I20260926 14:44:51.295493 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:44:51.307654 2844 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:44:51.306069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40816 (local address 127.0.98.62:45019)
0926 14:44:51.306190 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:51.306194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:51.306449 (+ 255us) server_negotiation.cc:407] Connection header received
0926 14:44:51.306556 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:51.306560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:51.306616 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:51.306784 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:51.307508 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40816: BlockingRecv error: recv got EOF from 127.0.0.1:40816 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:44:51.308908 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 2867
I20260926 14:44:51.313906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 3001
I20260926 14:44:51.318337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 3135
I20260926 14:44:51.322367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 2773
2026-09-26T14:44:51Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2041 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:44:51 dist-test-slave-s7k6 krb5kdc[3276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:44:51 dist-test-slave-s7k6 krb5kdc[3276](info): set up 2 sockets
Sep 26 14:44:51 dist-test-slave-s7k6 krb5kdc[3276](info): commencing operation
krb5kdc: starting...
W20260926 14:44:52.294516 392 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:44:52 dist-test-slave-s7k6 krb5kdc[3276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:44:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:44:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:44:52.410063 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34691
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:52.519397 3289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:52.519680 3289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:52.519721 3289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:52.523715 3289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:44:52.523767 3289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:52.523785 3289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:52.523806 3289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:44:52.523828 3289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:44:52.528986 3289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:52.529994 3289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:52.531003 3289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:52.539271 3295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:52.539309 3297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:52.539274 3294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:52.539327 3289 server_base.cc:1034] running on GCE node
I20260926 14:44:52.540033 3289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:52.541065 3289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:52.542236 3289 hybrid_clock.cc:648] HybridClock initialized: now 1790433892542214 us; error 36 us; skew 500 ppm
Sep 26 14:44:52 dist-test-slave-s7k6 krb5kdc[3276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:52.545037 3289 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:44:52.545898 3289 webserver.cc:460] Webserver started at http://127.0.98.62:39715/ using document root <none> and password file <none>
I20260926 14:44:52.546446 3289 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:52.546514 3289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:52.546698 3289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:52.548771 3289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "43d1840d72984f69aa765c6c5dc02527"
format_stamp: "Formatted at 2026-09-26 14:44:52 on dist-test-slave-s7k6"
server_key: "81b5ddc604bcf0d94e0a0108ae45158b"
server_key_iv: "a2e47038dcc83d08c999ca8d2bdc6ee1"
server_key_version: "encryptionkey@0"
I20260926 14:44:52.549335 3289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "43d1840d72984f69aa765c6c5dc02527"
format_stamp: "Formatted at 2026-09-26 14:44:52 on dist-test-slave-s7k6"
server_key: "81b5ddc604bcf0d94e0a0108ae45158b"
server_key_iv: "a2e47038dcc83d08c999ca8d2bdc6ee1"
server_key_version: "encryptionkey@0"
I20260926 14:44:52.553304 3289 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:44:52.555855 3304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:52.557152 3289 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:52.557292 3289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "43d1840d72984f69aa765c6c5dc02527"
format_stamp: "Formatted at 2026-09-26 14:44:52 on dist-test-slave-s7k6"
server_key: "81b5ddc604bcf0d94e0a0108ae45158b"
server_key_iv: "a2e47038dcc83d08c999ca8d2bdc6ee1"
server_key_version: "encryptionkey@0"
I20260926 14:44:52.557399 3289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:52.604555 3289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:52.608001 3289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:52.608196 3289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:52.615350 3289 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34691
I20260926 14:44:52.615653 3356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34691 every 8 connection(s)
I20260926 14:44:52.616338 3289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:44:52.619239 3357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:52.624739 3357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527: Bootstrap starting.
I20260926 14:44:52.626631 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 3289
I20260926 14:44:52.626747 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:44:52.627041 392 external_mini_cluster.cc:1428] Setting key ab9ff7ec2e96daf364202b22846f3fa1
I20260926 14:44:52.627475 3357 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:52.628345 3357 log.cc:824] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:52.630342 3357 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527: No bootstrap required, opened a new log
I20260926 14:44:52.633945 3357 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d1840d72984f69aa765c6c5dc02527" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34691 } }
I20260926 14:44:52.634254 3357 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:52.634317 3357 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43d1840d72984f69aa765c6c5dc02527, State: Initialized, Role: FOLLOWER
I20260926 14:44:52.634776 3357 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d1840d72984f69aa765c6c5dc02527" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34691 } }
I20260926 14:44:52.634922 3357 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:44:52.634965 3357 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:44:52.635062 3357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 14:44:52 dist-test-slave-s7k6 krb5kdc[3276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:52.636228 3357 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d1840d72984f69aa765c6c5dc02527" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34691 } }
I20260926 14:44:52.636617 3357 leader_election.cc:302] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43d1840d72984f69aa765c6c5dc02527; no voters:
I20260926 14:44:52.636966 3357 leader_election.cc:288] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:44:52.637066 3362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:52.637277 3362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [term 1 LEADER]: Becoming Leader. State: Replica: 43d1840d72984f69aa765c6c5dc02527, State: Running, Role: LEADER
I20260926 14:44:52.637686 3362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d1840d72984f69aa765c6c5dc02527" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34691 } }
I20260926 14:44:52.638249 3357 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:44:52.639469 3363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43d1840d72984f69aa765c6c5dc02527. Latest consensus state: current_term: 1 leader_uuid: "43d1840d72984f69aa765c6c5dc02527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d1840d72984f69aa765c6c5dc02527" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34691 } } }
I20260926 14:44:52.639688 3363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:52.639658 3364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43d1840d72984f69aa765c6c5dc02527" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43d1840d72984f69aa765c6c5dc02527" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34691 } } }
I20260926 14:44:52.639803 3364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:52.642465 3360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:52.628214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.0.98.62:34691)
0926 14:44:52.628679 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:52.628699 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:52.628747 (+ 48us) server_negotiation.cc:407] Connection header received
0926 14:44:52.629570 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:52.629592 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:52.629955 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:52.630288 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:52.631127 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:52.632147 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:52.633019 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:52.633508 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:52.633702 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:52.636609 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:52.636615 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:52.636629 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:52.636664 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:52.638833 (+ 2169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:52.639348 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:52.639355 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:52.639362 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:52.639518 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:52.640010 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:52.640014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:52.640016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:52.640533 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:52.640727 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:52.641158 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 14:44:52.641489 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":149,"threads_started":1}
W20260926 14:44:52.642900 3377 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:44:52.642961 3377 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:44:52.643049 3375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:44:52.643739 3375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:44:52.645671 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:44:52.649663 3375 catalog_manager.cc:1347] Generated new cluster ID: 0edfebf2e7134deca2e1c520a8073f12
I20260926 14:44:52.649724 3375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:44:52.665647 3360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:52.660075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.0.98.62:34691)
0926 14:44:52.660188 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:52.660192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:52.660419 (+ 227us) server_negotiation.cc:407] Connection header received
0926 14:44:52.660532 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:52.660535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:52.660584 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:52.660670 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:52.661265 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:52.662086 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:52.663274 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:52.663651 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:52.663752 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:52.664611 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:52.664615 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:52.664618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:44:52.665200 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:52.665237 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:44:52.665257 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:52.665343 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:52.665418 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 14:44:52.665485 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:44:52.676393 3375 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:44:52.677394 3375 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:44:52.685734 3375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43d1840d72984f69aa765c6c5dc02527: Generated new TSK 0
I20260926 14:44:52.686470 3375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:44:52.697197 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 3289
2026-09-26T14:44:52Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1374 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:44:52 dist-test-slave-s7k6 krb5kdc[3390](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:44:52 dist-test-slave-s7k6 krb5kdc[3390](info): set up 2 sockets
Sep 26 14:44:52 dist-test-slave-s7k6 krb5kdc[3390](info): commencing operation
krb5kdc: starting...
W20260926 14:44:53.646624 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:44:53 dist-test-slave-s7k6 krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:44:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:44:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:44:53.762187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38791
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38791
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:53.872328 3403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:53.872632 3403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:53.872674 3403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:53.876780 3403 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:44:53.876857 3403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:53.876874 3403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:53.876890 3403 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:44:53.876905 3403 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:44:53.882416 3403 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38791
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:53.883533 3403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:53.884552 3403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:53.892758 3408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:53.892793 3409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:53.892793 3411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:53.893122 3403 server_base.cc:1034] running on GCE node
I20260926 14:44:53.893673 3403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:53.894743 3403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:53.895927 3403 hybrid_clock.cc:648] HybridClock initialized: now 1790433893895907 us; error 35 us; skew 500 ppm
Sep 26 14:44:53 dist-test-slave-s7k6 krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:53.898479 3403 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:44:53.899385 3403 webserver.cc:460] Webserver started at http://127.0.98.62:33057/ using document root <none> and password file <none>
I20260926 14:44:53.899923 3403 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:53.899988 3403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:53.900170 3403 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:53.902149 3403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "072011872d304f1bbe5395694f601c6e"
format_stamp: "Formatted at 2026-09-26 14:44:53 on dist-test-slave-s7k6"
server_key: "a0a5fac44fd2803d293329bfba1fbd19"
server_key_iv: "79b0b15f35bce9b21aa97f549d52ab46"
server_key_version: "encryptionkey@0"
I20260926 14:44:53.902664 3403 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "072011872d304f1bbe5395694f601c6e"
format_stamp: "Formatted at 2026-09-26 14:44:53 on dist-test-slave-s7k6"
server_key: "a0a5fac44fd2803d293329bfba1fbd19"
server_key_iv: "79b0b15f35bce9b21aa97f549d52ab46"
server_key_version: "encryptionkey@0"
I20260926 14:44:53.906237 3403 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 14:44:53.908730 3418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:53.910216 3403 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:44:53.910365 3403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "072011872d304f1bbe5395694f601c6e"
format_stamp: "Formatted at 2026-09-26 14:44:53 on dist-test-slave-s7k6"
server_key: "a0a5fac44fd2803d293329bfba1fbd19"
server_key_iv: "79b0b15f35bce9b21aa97f549d52ab46"
server_key_version: "encryptionkey@0"
I20260926 14:44:53.910486 3403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:53.922272 3403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:53.925264 3403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:53.925443 3403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:53.932372 3403 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38791
I20260926 14:44:53.932623 3470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38791 every 8 connection(s)
I20260926 14:44:53.933375 3403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:44:53.936203 3471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:53.938033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 3403
I20260926 14:44:53.938143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:44:53.938390 392 external_mini_cluster.cc:1428] Setting key 8a8fd0ee65f8aa170319039590359733
I20260926 14:44:53.943023 3471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e: Bootstrap starting.
I20260926 14:44:53.945621 3471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:53.946453 3471 log.cc:824] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:44:53 dist-test-slave-s7k6 krb5kdc[3390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:53.948488 3471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e: No bootstrap required, opened a new log
I20260926 14:44:53.953032 3471 raft_consensus.cc:348] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072011872d304f1bbe5395694f601c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } }
I20260926 14:44:53.953312 3471 raft_consensus.cc:374] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:53.953375 3471 raft_consensus.cc:729] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072011872d304f1bbe5395694f601c6e, State: Initialized, Role: FOLLOWER
I20260926 14:44:53.953758 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:53.939884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40036 (local address 127.0.98.62:38791)
0926 14:44:53.940318 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:53.940337 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:53.940379 (+ 42us) server_negotiation.cc:407] Connection header received
0926 14:44:53.941121 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:53.941145 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:53.941577 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:53.941957 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:53.942644 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:53.944105 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:53.944957 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:53.945636 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:53.945797 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:53.948556 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:53.948562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:53.948579 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:53.948608 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:53.950498 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:53.951002 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:53.951008 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:53.951014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:53.951171 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:53.951418 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:53.951421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:53.951423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:53.951953 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:53.952163 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:53.952435 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 14:44:53.952745 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260926 14:44:53.953871 3471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072011872d304f1bbe5395694f601c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } }
I20260926 14:44:53.954022 3471 raft_consensus.cc:388] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:44:53.954087 3471 raft_consensus.cc:482] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:44:53.954180 3471 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:53.955243 3471 raft_consensus.cc:504] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072011872d304f1bbe5395694f601c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } }
I20260926 14:44:53.955621 3471 leader_election.cc:302] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 072011872d304f1bbe5395694f601c6e; no voters:
I20260926 14:44:53.955966 3471 leader_election.cc:288] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:44:53.956099 3476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:53.956403 3476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [term 1 LEADER]: Becoming Leader. State: Replica: 072011872d304f1bbe5395694f601c6e, State: Running, Role: LEADER
I20260926 14:44:53.956777 3476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072011872d304f1bbe5395694f601c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } }
I20260926 14:44:53.957158 3471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:44:53.958274 3477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "072011872d304f1bbe5395694f601c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072011872d304f1bbe5395694f601c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } } }
I20260926 14:44:53.958400 3477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:53.958803 3478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 072011872d304f1bbe5395694f601c6e. Latest consensus state: current_term: 1 leader_uuid: "072011872d304f1bbe5395694f601c6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "072011872d304f1bbe5395694f601c6e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38791 } } }
I20260926 14:44:53.958896 3478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e [sys.catalog]: This master's current role is: LEADER
W20260926 14:44:53.961599 3491 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:44:53.961665 3491 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:44:53.961720 3486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:44:53.962416 3486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:44:53.968339 3486 catalog_manager.cc:1347] Generated new cluster ID: dae320ebfe1647898113a0adc45996d5
I20260926 14:44:53.968400 3486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:44:53.980638 3486 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:44:53.981581 3486 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:44:53.995055 3486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 072011872d304f1bbe5395694f601c6e: Generated new TSK 0
I20260926 14:44:53.996035 3486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:44:54.031543 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38791
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:54.141182 3497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:54.141453 3497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:54.141512 3497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:54.145670 3497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:54.145738 3497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:54.145820 3497 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:44:54.151191 3497 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3497
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:54.152318 3497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:54.153385 3497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:54.162968 3505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:54.163000 3503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:54.163033 3502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:54.163239 3497 server_base.cc:1034] running on GCE node
I20260926 14:44:54.163669 3497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:54.164294 3497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:54.165462 3497 hybrid_clock.cc:648] HybridClock initialized: now 1790433894165445 us; error 33 us; skew 500 ppm
Sep 26 14:44:54 dist-test-slave-s7k6 krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:54.168270 3497 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:44:54.169137 3497 webserver.cc:460] Webserver started at http://127.0.98.1:43009/ using document root <none> and password file <none>
I20260926 14:44:54.169689 3497 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:54.169755 3497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:54.169993 3497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:54.172080 3497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "6483e74fb56445a7ba8b7fdfb8466e2a"
format_stamp: "Formatted at 2026-09-26 14:44:54 on dist-test-slave-s7k6"
server_key: "0b5cfcfa119906f0fc27614ea6eba25c"
server_key_iv: "7f34bd3570a965e14e16a1ebf8195836"
server_key_version: "encryptionkey@0"
I20260926 14:44:54.172631 3497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6483e74fb56445a7ba8b7fdfb8466e2a"
format_stamp: "Formatted at 2026-09-26 14:44:54 on dist-test-slave-s7k6"
server_key: "0b5cfcfa119906f0fc27614ea6eba25c"
server_key_iv: "7f34bd3570a965e14e16a1ebf8195836"
server_key_version: "encryptionkey@0"
I20260926 14:44:54.176303 3497 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:44:54.178957 3512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:54.180264 3497 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:54.180377 3497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "6483e74fb56445a7ba8b7fdfb8466e2a"
format_stamp: "Formatted at 2026-09-26 14:44:54 on dist-test-slave-s7k6"
server_key: "0b5cfcfa119906f0fc27614ea6eba25c"
server_key_iv: "7f34bd3570a965e14e16a1ebf8195836"
server_key_version: "encryptionkey@0"
I20260926 14:44:54.180492 3497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:54.200412 3497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:54.203915 3497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:54.204111 3497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:54.204718 3497 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:54.205561 3497 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:54.205610 3497 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:54.205662 3497 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:54.205687 3497 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:54.214020 3497 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36249
I20260926 14:44:54.214028 3625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36249 every 8 connection(s)
I20260926 14:44:54.215046 3497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:44:54.218113 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 3497
I20260926 14:44:54.218216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:44:54.218468 392 external_mini_cluster.cc:1428] Setting key 2176d6d03bb32cdad60d4b648cc18876
Sep 26 14:44:54 dist-test-slave-s7k6 krb5kdc[3390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 14:44:54.229370 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:54.216831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46667 (local address 127.0.98.62:38791)
0926 14:44:54.216972 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:54.216975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:54.217832 (+ 857us) server_negotiation.cc:407] Connection header received
0926 14:44:54.219411 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:54.219414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:54.219454 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:54.219516 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:54.220785 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.221491 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.222724 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.222994 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.223102 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:54.225763 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:54.225767 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:54.225769 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:54.225793 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:54.227231 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:54.227732 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:54.227735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:54.227737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:54.227831 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:54.228211 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:54.228215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:54.228216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:54.228417 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:54.228519 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:54.229072 (+ 553us) server_negotiation.cc:299] Negotiation successful
0926 14:44:54.229193 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 14:44:54.230315 3628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:54.217233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38791 (local address 127.0.98.1:46667)
0926 14:44:54.217684 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:54.217726 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:54.219212 (+ 1486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:54.219694 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:54.219704 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:54.220137 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:54.220631 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:54.220646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.221677 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.221681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:54.222603 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:54.222613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.223141 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.223145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:54.223323 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:54.223872 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:54.223888 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:54.225577 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:54.227351 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:54.227356 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:54.227369 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:54.227640 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:54.227909 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:54.227911 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:54.227913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:54.228092 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:54.228534 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:54.228540 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:54.228806 (+ 266us) client_negotiation.cc:769] Sending connection context
0926 14:44:54.228979 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 14:44:54.229309 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260926 14:44:54.231729 3626 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38791
I20260926 14:44:54.231971 3626 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:54.232496 3626 heartbeater.cc:511] Master 127.0.98.62:38791 requested a full tablet report, sending...
I20260926 14:44:54.234220 3435 ts_manager.cc:194] Registered new tserver with Master: 6483e74fb56445a7ba8b7fdfb8466e2a (127.0.98.1:36249)
I20260926 14:44:54.235875 3435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46667
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:44:54.246102 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38791
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:54.361223 3631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:54.361479 3631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:54.361522 3631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:54.365640 3631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:54.365706 3631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:54.365765 3631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:44:54.371183 3631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:54.372221 3631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:54.373219 3631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:54.382977 3639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:54.383069 3636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:54.383112 3637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:54.383155 3631 server_base.cc:1034] running on GCE node
I20260926 14:44:54.383671 3631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:54.384348 3631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:54.385532 3631 hybrid_clock.cc:648] HybridClock initialized: now 1790433894385507 us; error 41 us; skew 500 ppm
Sep 26 14:44:54 dist-test-slave-s7k6 krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:54.388296 3631 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:44:54.389200 3631 webserver.cc:460] Webserver started at http://127.0.98.2:41487/ using document root <none> and password file <none>
I20260926 14:44:54.389762 3631 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:54.389832 3631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:54.390022 3631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:54.392239 3631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "956e8613894c4da5985638591d2eaff1"
format_stamp: "Formatted at 2026-09-26 14:44:54 on dist-test-slave-s7k6"
server_key: "5a42863d2e932f37c891ac27552f8cf3"
server_key_iv: "57eb2714767b0e22882e8d71cb90bd7c"
server_key_version: "encryptionkey@0"
I20260926 14:44:54.392792 3631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "956e8613894c4da5985638591d2eaff1"
format_stamp: "Formatted at 2026-09-26 14:44:54 on dist-test-slave-s7k6"
server_key: "5a42863d2e932f37c891ac27552f8cf3"
server_key_iv: "57eb2714767b0e22882e8d71cb90bd7c"
server_key_version: "encryptionkey@0"
I20260926 14:44:54.396854 3631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:44:54.400249 3646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:54.402002 3631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:44:54.402154 3631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "956e8613894c4da5985638591d2eaff1"
format_stamp: "Formatted at 2026-09-26 14:44:54 on dist-test-slave-s7k6"
server_key: "5a42863d2e932f37c891ac27552f8cf3"
server_key_iv: "57eb2714767b0e22882e8d71cb90bd7c"
server_key_version: "encryptionkey@0"
I20260926 14:44:54.402293 3631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:54.421099 3631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:54.424393 3631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:54.424571 3631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:54.425143 3631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:54.426003 3631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:54.426050 3631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:54.426092 3631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:54.426122 3631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:54.434731 3631 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41897
I20260926 14:44:54.434808 3759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41897 every 8 connection(s)
I20260926 14:44:54.435702 3631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:44:54.442476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 3631
I20260926 14:44:54.442587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:44:54.442914 392 external_mini_cluster.cc:1428] Setting key 7068ac1704b9051de2bb860d7f05a6d9
Sep 26 14:44:54 dist-test-slave-s7k6 krb5kdc[3390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:54.449883 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:54.437631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55231 (local address 127.0.98.62:38791)
0926 14:44:54.437770 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:54.437774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:54.438678 (+ 904us) server_negotiation.cc:407] Connection header received
0926 14:44:54.440185 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:54.440189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:54.440249 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:54.440321 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:54.441580 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.442132 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.443426 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.443713 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.443806 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:54.446531 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:54.446535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:54.446537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:54.446559 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:54.447873 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:54.448383 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:54.448386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:54.448387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:54.448461 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:54.448787 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:54.448791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:54.448792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:54.448982 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:54.449069 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:54.449586 (+ 517us) server_negotiation.cc:299] Negotiation successful
0926 14:44:54.449713 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":53}
I20260926 14:44:54.450722 3762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:54.438027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38791 (local address 127.0.98.2:55231)
0926 14:44:54.438496 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:54.438540 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:54.439931 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:54.440516 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:54.440524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:54.440911 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:54.441423 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:54.441436 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.442261 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.442264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:54.443304 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:54.443315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.443862 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.443867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:54.444053 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:54.444752 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:54.444769 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:54.446346 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:54.447993 (+ 1647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:54.447999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:54.448017 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:54.448284 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:54.448551 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:54.448555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:54.448557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:54.448700 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:54.449080 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:54.449088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:54.449343 (+ 255us) client_negotiation.cc:769] Sending connection context
0926 14:44:54.449529 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 14:44:54.449870 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":118,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 14:44:54.451951 3760 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38791
I20260926 14:44:54.452234 3760 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:54.452760 3760 heartbeater.cc:511] Master 127.0.98.62:38791 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:44:54.453890 3435 ts_manager.cc:194] Registered new tserver with Master: 956e8613894c4da5985638591d2eaff1 (127.0.98.2:41897)
I20260926 14:44:54.454658 3435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:44:54.469101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38791
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:54.579854 3765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:54.580142 3765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:54.580185 3765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:54.584275 3765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:54.584335 3765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:54.584403 3765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:44:54.589792 3765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3765
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:54.590955 3765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:54.592056 3765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:54.602169 3771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:54.602192 3770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:54.602169 3773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:54.602286 3765 server_base.cc:1034] running on GCE node
I20260926 14:44:54.602883 3765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:54.603590 3765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:54.604772 3765 hybrid_clock.cc:648] HybridClock initialized: now 1790433894604753 us; error 33 us; skew 500 ppm
Sep 26 14:44:54 dist-test-slave-s7k6 krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:54.607498 3765 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:44:54.608422 3765 webserver.cc:460] Webserver started at http://127.0.98.3:39831/ using document root <none> and password file <none>
I20260926 14:44:54.608960 3765 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:54.609025 3765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:54.609218 3765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:54.611253 3765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "a62ceee2fa184d87a507196fe0ee8566"
format_stamp: "Formatted at 2026-09-26 14:44:54 on dist-test-slave-s7k6"
server_key: "49e2c264f6bfbde3814de8a6095d85f9"
server_key_iv: "6465b2a791cd5bac3d8d4491ed7fd691"
server_key_version: "encryptionkey@0"
I20260926 14:44:54.611783 3765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a62ceee2fa184d87a507196fe0ee8566"
format_stamp: "Formatted at 2026-09-26 14:44:54 on dist-test-slave-s7k6"
server_key: "49e2c264f6bfbde3814de8a6095d85f9"
server_key_iv: "6465b2a791cd5bac3d8d4491ed7fd691"
server_key_version: "encryptionkey@0"
I20260926 14:44:54.615725 3765 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:44:54.618153 3780 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:54.619482 3765 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:54.619586 3765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "a62ceee2fa184d87a507196fe0ee8566"
format_stamp: "Formatted at 2026-09-26 14:44:54 on dist-test-slave-s7k6"
server_key: "49e2c264f6bfbde3814de8a6095d85f9"
server_key_iv: "6465b2a791cd5bac3d8d4491ed7fd691"
server_key_version: "encryptionkey@0"
I20260926 14:44:54.619712 3765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:54.642344 3765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:54.645673 3765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:54.645846 3765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:54.646443 3765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:54.647297 3765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:54.647346 3765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:54.647385 3765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:54.647410 3765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:54.655380 3765 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34195
I20260926 14:44:54.655445 3893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34195 every 8 connection(s)
I20260926 14:44:54.656265 3765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:44:54.665467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 3765
I20260926 14:44:54.665612 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
Sep 26 14:44:54 dist-test-slave-s7k6 krb5kdc[3390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:54.665928 392 external_mini_cluster.cc:1428] Setting key 63c8e84edc9597c9ab67c28c2377afd3
I20260926 14:44:54.670166 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:54.658033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50115 (local address 127.0.98.62:38791)
0926 14:44:54.658246 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:54.658250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:54.659143 (+ 893us) server_negotiation.cc:407] Connection header received
0926 14:44:54.660614 (+ 1471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:54.660618 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:54.660671 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:54.660747 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:54.661902 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.662478 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.663607 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.663827 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.663908 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:54.666473 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:54.666478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:54.666480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:54.666507 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:54.667997 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:54.668493 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:54.668496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:54.668498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:54.668590 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:54.668967 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:54.668970 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:54.668972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:54.669213 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:54.669312 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:54.669900 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 14:44:54.670023 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:44:54.671159 3896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:54.658509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38791 (local address 127.0.98.3:50115)
0926 14:44:54.658991 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:54.659036 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:54.660355 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:54.660905 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:54.660913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:54.661283 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:54.661777 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:54.661792 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.662586 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.662588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:54.663501 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:54.663511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.663945 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.663948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:54.664091 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:54.664614 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:54.664629 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:54.666278 (+ 1649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:54.668095 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:54.668101 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:54.668119 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:54.668387 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:54.668712 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:54.668715 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:54.668716 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:54.668861 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:54.669339 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:54.669345 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:54.669615 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 14:44:54.669810 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 14:44:54.670128 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":121,"threads_started":1}
I20260926 14:44:54.672420 3894 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38791
I20260926 14:44:54.672657 3894 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:54.673158 3894 heartbeater.cc:511] Master 127.0.98.62:38791 requested a full tablet report, sending...
I20260926 14:44:54.674257 3435 ts_manager.cc:194] Registered new tserver with Master: a62ceee2fa184d87a507196fe0ee8566 (127.0.98.3:34195)
I20260926 14:44:54.674934 3435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50115
I20260926 14:44:54.680079 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:44:54 dist-test-slave-s7k6 krb5kdc[3390](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433894, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 14:44:54 dist-test-slave-s7k6 krb5kdc[3390](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433894, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:54.704023 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:54.696241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58160 (local address 127.0.98.62:38791)
0926 14:44:54.696389 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:54.696394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:54.696515 (+ 121us) server_negotiation.cc:407] Connection header received
0926 14:44:54.696662 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:54.696666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:54.696713 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:54.696781 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:54.697449 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.697964 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.698834 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:54.699051 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:54.699155 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:54.701371 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:54.701375 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:54.701377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:54.701401 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:54.702557 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:54.703005 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:54.703008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:54.703009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:54.703081 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:54.703350 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:54.703352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:54.703354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:54.703559 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:54.703667 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:54.703814 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 14:44:54.703906 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 14:44:54.704380 3435 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58160
I20260926 14:44:54.705720 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 3497
I20260926 14:44:54.711611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 3631
I20260926 14:44:54.716948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 3765
I20260926 14:44:54.722606 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 3403
2026-09-26T14:44:54Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2026 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:44:54 dist-test-slave-s7k6 krb5kdc[3906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:44:54 dist-test-slave-s7k6 krb5kdc[3906](info): set up 2 sockets
Sep 26 14:44:54 dist-test-slave-s7k6 krb5kdc[3906](info): commencing operation
krb5kdc: starting...
W20260926 14:44:55.678606 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:44:55 dist-test-slave-s7k6 krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:44:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:44:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:44:55.797163 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40543
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:55.903975 3919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:55.904218 3919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:55.904254 3919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:55.908522 3919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:44:55.908581 3919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:55.908598 3919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:55.908613 3919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:44:55.908630 3919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:44:55.914304 3919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:55.915419 3919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:55.916446 3919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:44:55.924569 3919 server_base.cc:1034] running on GCE node
W20260926 14:44:55.924541 3924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:55.924527 3925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:55.924527 3927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:55.925570 3919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:55.926707 3919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:55.927894 3919 hybrid_clock.cc:648] HybridClock initialized: now 1790433895927876 us; error 31 us; skew 500 ppm
Sep 26 14:44:55 dist-test-slave-s7k6 krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433895, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:55.937191 3919 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:44:55.938054 3919 webserver.cc:460] Webserver started at http://127.0.98.62:36581/ using document root <none> and password file <none>
I20260926 14:44:55.938556 3919 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:55.938611 3919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:55.938807 3919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:55.940775 3919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "bcb650eefe774411a3c612e852815e79"
format_stamp: "Formatted at 2026-09-26 14:44:55 on dist-test-slave-s7k6"
server_key: "3d7bc58e9ad8846f3a736aefe8291b2f"
server_key_iv: "fb43a3e442023fbdd9ebe1e0b1922568"
server_key_version: "encryptionkey@0"
I20260926 14:44:55.941250 3919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "bcb650eefe774411a3c612e852815e79"
format_stamp: "Formatted at 2026-09-26 14:44:55 on dist-test-slave-s7k6"
server_key: "3d7bc58e9ad8846f3a736aefe8291b2f"
server_key_iv: "fb43a3e442023fbdd9ebe1e0b1922568"
server_key_version: "encryptionkey@0"
I20260926 14:44:55.944761 3919 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:44:55.947032 3934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:55.948230 3919 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:44:55.948340 3919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "bcb650eefe774411a3c612e852815e79"
format_stamp: "Formatted at 2026-09-26 14:44:55 on dist-test-slave-s7k6"
server_key: "3d7bc58e9ad8846f3a736aefe8291b2f"
server_key_iv: "fb43a3e442023fbdd9ebe1e0b1922568"
server_key_version: "encryptionkey@0"
I20260926 14:44:55.948429 3919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:55.962729 3919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:55.965873 3919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:55.966050 3919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:55.972661 3919 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40543
I20260926 14:44:55.972683 3986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40543 every 8 connection(s)
I20260926 14:44:55.973582 3919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:44:55.976434 3987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:55.981855 3987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79: Bootstrap starting.
I20260926 14:44:55.983263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 3919
I20260926 14:44:55.983384 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:44:55.983618 392 external_mini_cluster.cc:1428] Setting key 1751efa4b0f2ae45105940c5c2033105
I20260926 14:44:55.984552 3987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:55.985469 3987 log.cc:824] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:55.987681 3987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79: No bootstrap required, opened a new log
I20260926 14:44:55.991812 3987 raft_consensus.cc:348] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb650eefe774411a3c612e852815e79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260926 14:44:55.992097 3987 raft_consensus.cc:374] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:55.992157 3987 raft_consensus.cc:729] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcb650eefe774411a3c612e852815e79, State: Initialized, Role: FOLLOWER
Sep 26 14:44:55 dist-test-slave-s7k6 krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:55.992630 3987 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb650eefe774411a3c612e852815e79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260926 14:44:55.992755 3987 raft_consensus.cc:388] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:44:55.992815 3987 raft_consensus.cc:482] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:44:55.992910 3987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:55.993932 3987 raft_consensus.cc:504] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb650eefe774411a3c612e852815e79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260926 14:44:55.994279 3987 leader_election.cc:302] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcb650eefe774411a3c612e852815e79; no voters:
I20260926 14:44:55.994581 3987 leader_election.cc:288] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:44:55.994694 3992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:55.994943 3992 raft_consensus.cc:686] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [term 1 LEADER]: Becoming Leader. State: Replica: bcb650eefe774411a3c612e852815e79, State: Running, Role: LEADER
I20260926 14:44:55.995292 3992 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb650eefe774411a3c612e852815e79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } }
I20260926 14:44:55.995973 3987 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:44:55.996899 3994 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcb650eefe774411a3c612e852815e79. Latest consensus state: current_term: 1 leader_uuid: "bcb650eefe774411a3c612e852815e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb650eefe774411a3c612e852815e79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } } }
I20260926 14:44:55.997247 3994 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:55.997144 3993 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcb650eefe774411a3c612e852815e79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcb650eefe774411a3c612e852815e79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40543 } } }
I20260926 14:44:55.997532 3993 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79 [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:55.997823 4001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:44:55.998735 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:55.984695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.0.98.62:40543)
0926 14:44:55.985149 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:55.985164 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:55.985197 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:44:55.985872 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:55.985897 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:55.986290 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:55.986571 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:55.987480 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:55.988477 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:55.989617 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:55.990099 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:55.990209 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:55.993046 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:55.993051 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:55.993065 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:55.993094 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:55.994967 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:55.995474 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:55.995479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:55.995485 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:55.995618 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:55.995959 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:55.995962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:55.995965 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:55.996536 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:55.996729 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:55.997466 (+ 737us) server_negotiation.cc:299] Negotiation successful
0926 14:44:55.997778 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":133,"threads_started":1}
I20260926 14:44:56.000874 4001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:44:56.006132 4001 catalog_manager.cc:1347] Generated new cluster ID: f4b2a7875d9542f0a75c754f5e7f1239
I20260926 14:44:56.006194 4001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:44:56.035356 4001 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:44:56.036544 4001 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:44:56.045612 4001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bcb650eefe774411a3c612e852815e79: Generated new TSK 0
I20260926 14:44:56.046478 4001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:44:56.081327 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:56.191139 4013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:56.191465 4013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:56.191540 4013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:56.195731 4013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:56.195796 4013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:56.195875 4013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:44:56.201493 4013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:56.202595 4013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:56.203648 4013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:56.213166 4021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:56.213166 4019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:56.213191 4018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:56.213515 4013 server_base.cc:1034] running on GCE node
I20260926 14:44:56.213937 4013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:56.214584 4013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:56.215766 4013 hybrid_clock.cc:648] HybridClock initialized: now 1790433896215742 us; error 39 us; skew 500 ppm
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:56.218690 4013 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:44:56.219579 4013 webserver.cc:460] Webserver started at http://127.0.98.1:41823/ using document root <none> and password file <none>
I20260926 14:44:56.220090 4013 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:56.220147 4013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:56.220336 4013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:56.222345 4013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "a9dfd1ddd09e462c82fabd0bf81ab5cf"
format_stamp: "Formatted at 2026-09-26 14:44:56 on dist-test-slave-s7k6"
server_key: "355f3615e854353275bcff47f0b9ed91"
server_key_iv: "bd064323596c5091646e8384b5e18a6c"
server_key_version: "encryptionkey@0"
I20260926 14:44:56.222910 4013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a9dfd1ddd09e462c82fabd0bf81ab5cf"
format_stamp: "Formatted at 2026-09-26 14:44:56 on dist-test-slave-s7k6"
server_key: "355f3615e854353275bcff47f0b9ed91"
server_key_iv: "bd064323596c5091646e8384b5e18a6c"
server_key_version: "encryptionkey@0"
I20260926 14:44:56.226632 4013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:44:56.229287 4028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:56.230675 4013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:44:56.230835 4013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "a9dfd1ddd09e462c82fabd0bf81ab5cf"
format_stamp: "Formatted at 2026-09-26 14:44:56 on dist-test-slave-s7k6"
server_key: "355f3615e854353275bcff47f0b9ed91"
server_key_iv: "bd064323596c5091646e8384b5e18a6c"
server_key_version: "encryptionkey@0"
I20260926 14:44:56.230952 4013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:56.271217 4013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:56.274807 4013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:56.274976 4013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:56.275539 4013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:56.276346 4013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:56.276393 4013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:56.276433 4013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:56.276448 4013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:56.284654 4013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35945
I20260926 14:44:56.284690 4141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35945 every 8 connection(s)
I20260926 14:44:56.285558 4013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:44:56.288014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 4013
I20260926 14:44:56.288152 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:44:56.288475 392 external_mini_cluster.cc:1428] Setting key 1f751c3fc27e1f185f96d56dda93c7bb
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 14:44:56.300572 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:56.287577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40655 (local address 127.0.98.62:40543)
0926 14:44:56.287729 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:56.287733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:56.288778 (+ 1045us) server_negotiation.cc:407] Connection header received
0926 14:44:56.290428 (+ 1650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:56.290432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:56.290485 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:56.290557 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:56.291819 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.292344 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.293483 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.293726 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.293831 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:56.296627 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:56.296631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:56.296633 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:56.296685 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:56.298223 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:56.298811 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:56.298815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:56.298816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:56.298937 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:56.299347 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:56.299363 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:56.299365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:56.299560 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:56.299670 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:56.300252 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 14:44:56.300407 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 14:44:56.301391 4144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:56.288072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.1:40655)
0926 14:44:56.288627 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:56.288673 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:56.290209 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:56.290749 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:56.290785 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:56.291157 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:56.291670 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:56.291684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.292461 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.292464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:56.293364 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:56.293375 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.293880 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.293884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:56.294053 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:56.294684 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:56.294700 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:56.296429 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:56.298334 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:56.298340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:56.298354 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:56.298623 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:56.299054 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:56.299057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:56.299058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:56.299215 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:56.299674 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:56.299681 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:56.299991 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 14:44:56.300165 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 14:44:56.300475 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":167,"threads_started":1}
I20260926 14:44:56.302630 4142 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40543
I20260926 14:44:56.302899 4142 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:56.303419 4142 heartbeater.cc:511] Master 127.0.98.62:40543 requested a full tablet report, sending...
I20260926 14:44:56.305037 3951 ts_manager.cc:194] Registered new tserver with Master: a9dfd1ddd09e462c82fabd0bf81ab5cf (127.0.98.1:35945)
I20260926 14:44:56.306195 3951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:44:56.315956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:56.427755 4147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:56.428042 4147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:56.428083 4147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:56.432169 4147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:56.432241 4147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:56.432303 4147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:44:56.437776 4147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:56.438872 4147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:56.439858 4147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:56.449179 4153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:56.449160 4155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:56.449155 4152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:56.449325 4147 server_base.cc:1034] running on GCE node
I20260926 14:44:56.449955 4147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:56.450620 4147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:56.451829 4147 hybrid_clock.cc:648] HybridClock initialized: now 1790433896451811 us; error 45 us; skew 500 ppm
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:56.454635 4147 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:44:56.455580 4147 webserver.cc:460] Webserver started at http://127.0.98.2:46453/ using document root <none> and password file <none>
I20260926 14:44:56.456107 4147 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:56.456166 4147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:56.456337 4147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:56.458346 4147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "a09153ca30f74fcc82affbcb0752b704"
format_stamp: "Formatted at 2026-09-26 14:44:56 on dist-test-slave-s7k6"
server_key: "20e56b999bbcf15d16894aacf4dee5b2"
server_key_iv: "fc0aadabad7bea68a05dbafd4fb83122"
server_key_version: "encryptionkey@0"
I20260926 14:44:56.458901 4147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a09153ca30f74fcc82affbcb0752b704"
format_stamp: "Formatted at 2026-09-26 14:44:56 on dist-test-slave-s7k6"
server_key: "20e56b999bbcf15d16894aacf4dee5b2"
server_key_iv: "fc0aadabad7bea68a05dbafd4fb83122"
server_key_version: "encryptionkey@0"
I20260926 14:44:56.462504 4147 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:44:56.464964 4162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:56.466128 4147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:44:56.466226 4147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "a09153ca30f74fcc82affbcb0752b704"
format_stamp: "Formatted at 2026-09-26 14:44:56 on dist-test-slave-s7k6"
server_key: "20e56b999bbcf15d16894aacf4dee5b2"
server_key_iv: "fc0aadabad7bea68a05dbafd4fb83122"
server_key_version: "encryptionkey@0"
I20260926 14:44:56.466329 4147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:56.506222 4147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:56.509402 4147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:56.509577 4147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:56.510161 4147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:56.511214 4147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:56.511276 4147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:56.511355 4147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:56.511387 4147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:56.520093 4147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32797
I20260926 14:44:56.520159 4275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32797 every 8 connection(s)
I20260926 14:44:56.521027 4147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:44:56.522526 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 4147
I20260926 14:44:56.522624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:44:56.522946 392 external_mini_cluster.cc:1428] Setting key 0acf41b3b196db773ca36086def4cf98
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:44:56.534401 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:56.522800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45855 (local address 127.0.98.62:40543)
0926 14:44:56.522956 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:56.522960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:56.523814 (+ 854us) server_negotiation.cc:407] Connection header received
0926 14:44:56.525157 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:56.525160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:56.525211 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:56.525283 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:56.526414 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.526978 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.528080 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.528284 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.528369 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:56.531009 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:56.531011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:56.531013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:56.531036 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:56.532217 (+ 1181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:56.532731 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:56.532734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:56.532736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:56.532810 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:56.533115 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:56.533118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:56.533120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:56.533375 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:56.533460 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:56.534047 (+ 587us) server_negotiation.cc:299] Negotiation successful
0926 14:44:56.534171 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:44:56.535315 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:56.523214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.2:45855)
0926 14:44:56.523662 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:56.523716 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:56.524966 (+ 1250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:56.525460 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:56.525467 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:56.525810 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:56.526274 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:56.526287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.527092 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.527095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:56.527963 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:56.527972 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.528413 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.528416 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:56.528568 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:56.529166 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:56.529182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:56.530849 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:56.532365 (+ 1516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:56.532372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:56.532387 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:56.532640 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:56.532891 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:56.532894 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:56.532896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:56.533028 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:56.533473 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:56.533480 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:56.533820 (+ 340us) client_negotiation.cc:769] Sending connection context
0926 14:44:56.533994 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 14:44:56.534374 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":114,"threads_started":1}
I20260926 14:44:56.536567 4276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40543
I20260926 14:44:56.536823 4276 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:56.537431 4276 heartbeater.cc:511] Master 127.0.98.62:40543 requested a full tablet report, sending...
I20260926 14:44:56.538589 3951 ts_manager.cc:194] Registered new tserver with Master: a09153ca30f74fcc82affbcb0752b704 (127.0.98.2:32797)
I20260926 14:44:56.539206 3951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:44:56.549464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:56.658090 4281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:56.658367 4281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:56.658427 4281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:56.662478 4281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:56.662549 4281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:56.662632 4281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:44:56.667905 4281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:56.668942 4281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:56.669960 4281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:56.679690 4289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:56.679708 4286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:56.679714 4287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:56.680065 4281 server_base.cc:1034] running on GCE node
I20260926 14:44:56.680478 4281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:56.681087 4281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:56.682231 4281 hybrid_clock.cc:648] HybridClock initialized: now 1790433896682198 us; error 48 us; skew 500 ppm
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:56.684950 4281 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:44:56.685789 4281 webserver.cc:460] Webserver started at http://127.0.98.3:36649/ using document root <none> and password file <none>
I20260926 14:44:56.686323 4281 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:56.686388 4281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:56.686569 4281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:56.688606 4281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "9aa20209cfc24dd5a9d12f64fc9b6f58"
format_stamp: "Formatted at 2026-09-26 14:44:56 on dist-test-slave-s7k6"
server_key: "5c5b2e3587cbe6a599626e9286b17700"
server_key_iv: "daeb4444f4b2fed93103e333c148ffef"
server_key_version: "encryptionkey@0"
I20260926 14:44:56.689116 4281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9aa20209cfc24dd5a9d12f64fc9b6f58"
format_stamp: "Formatted at 2026-09-26 14:44:56 on dist-test-slave-s7k6"
server_key: "5c5b2e3587cbe6a599626e9286b17700"
server_key_iv: "daeb4444f4b2fed93103e333c148ffef"
server_key_version: "encryptionkey@0"
I20260926 14:44:56.692806 4281 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:44:56.695324 4296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:56.696578 4281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:56.696719 4281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "9aa20209cfc24dd5a9d12f64fc9b6f58"
format_stamp: "Formatted at 2026-09-26 14:44:56 on dist-test-slave-s7k6"
server_key: "5c5b2e3587cbe6a599626e9286b17700"
server_key_iv: "daeb4444f4b2fed93103e333c148ffef"
server_key_version: "encryptionkey@0"
I20260926 14:44:56.696820 4281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:56.717947 4281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:56.721000 4281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:56.721181 4281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:56.721766 4281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:56.722565 4281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:56.722611 4281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:56.722653 4281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:56.722681 4281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:56.730559 4281 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42687
I20260926 14:44:56.730605 4409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42687 every 8 connection(s)
I20260926 14:44:56.731472 4281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:44:56.735797 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 4281
I20260926 14:44:56.735924 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:44:56.736217 392 external_mini_cluster.cc:1428] Setting key 7671041fade1cc8fb34844b8ac9b5d2a
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:56.744261 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:56.733122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42565 (local address 127.0.98.62:40543)
0926 14:44:56.733271 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:56.733275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:56.734120 (+ 845us) server_negotiation.cc:407] Connection header received
0926 14:44:56.735471 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:56.735474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:56.735522 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:56.735596 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:56.737048 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.737647 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.738904 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.739171 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.739249 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:56.741426 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:56.741429 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:56.741431 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:56.741453 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:56.742565 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:56.743036 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:56.743038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:56.743039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:56.743112 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:56.743374 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:56.743377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:56.743378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:56.743553 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:56.743628 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:56.744047 (+ 419us) server_negotiation.cc:299] Negotiation successful
0926 14:44:56.744143 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:44:56.745033 4412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:56.733561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40543 (local address 127.0.98.3:42565)
0926 14:44:56.733978 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:56.734020 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:56.735255 (+ 1235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:56.735805 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:56.735815 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:56.736347 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:56.736903 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:56.736923 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.737762 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.737766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:56.738742 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:56.738752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.739285 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.739288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:56.739431 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:56.739954 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:56.739969 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:56.741289 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:56.742650 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:56.742654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:56.742666 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:56.742956 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:56.743174 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:56.743176 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:56.743177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:56.743297 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:56.743625 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:56.743630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:56.743847 (+ 217us) client_negotiation.cc:769] Sending connection context
0926 14:44:56.744002 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 14:44:56.744242 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":113,"threads_started":1}
I20260926 14:44:56.746095 4410 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40543
I20260926 14:44:56.746295 4410 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:56.746774 4410 heartbeater.cc:511] Master 127.0.98.62:40543 requested a full tablet report, sending...
I20260926 14:44:56.747731 3951 ts_manager.cc:194] Registered new tserver with Master: 9aa20209cfc24dd5a9d12f64fc9b6f58 (127.0.98.3:42687)
I20260926 14:44:56.748245 3951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42565
I20260926 14:44:56.749775 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[3906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 14:44:56.774523 4417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:56.765118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37574 (local address 127.0.98.1:35945)
0926 14:44:56.765465 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:56.765479 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:56.765502 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:44:56.765552 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:56.765557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:56.765755 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:56.765875 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:56.766552 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.767371 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.768168 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.768561 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.768639 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:56.771141 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:56.771150 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:56.771165 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:56.771189 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:56.772806 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:56.773217 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:56.773223 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:56.773225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:56.773306 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:56.773551 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:56.773554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:56.773556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:56.773795 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:56.774009 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:56.774190 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 14:44:56.774340 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":154,"threads_started":1}
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[3906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:56.785056 3990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:56.777413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37670 (local address 127.0.98.62:40543)
0926 14:44:56.777548 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:56.777552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:56.777627 (+ 75us) server_negotiation.cc:407] Connection header received
0926 14:44:56.777758 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:56.777761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:56.777807 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:56.777871 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:56.778481 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.779057 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.779819 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:56.780035 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:56.780122 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:56.782358 (+ 2236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:56.782361 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:56.782363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:56.782413 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:56.783571 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:56.783929 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:56.783933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:56.783935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:56.784009 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:56.784235 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:56.784238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:56.784240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:56.784455 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:56.784575 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:56.784793 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 14:44:56.784906 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 14:44:56.785485 3951 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37670
I20260926 14:44:56.786058 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 4013
I20260926 14:44:56.791378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 4147
I20260926 14:44:56.796078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 4281
I20260926 14:44:56.800812 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 3919
2026-09-26T14:44:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2077 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[4422](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[4422](info): set up 2 sockets
Sep 26 14:44:56 dist-test-slave-s7k6 krb5kdc[4422](info): commencing operation
krb5kdc: starting...
W20260926 14:44:57.751036 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:44:57 dist-test-slave-s7k6 krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:44:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:44:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:44:57.869663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41195
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:57.984366 4435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:57.984712 4435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:57.984781 4435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:57.989179 4435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:44:57.989259 4435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:57.989277 4435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:57.989293 4435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:44:57.989308 4435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:44:57.995147 4435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:57.996315 4435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:57.997434 4435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:58.006443 4443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:58.006446 4440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:58.006500 4441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:58.006814 4435 server_base.cc:1034] running on GCE node
I20260926 14:44:58.007445 4435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:58.008664 4435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:58.009847 4435 hybrid_clock.cc:648] HybridClock initialized: now 1790433898009822 us; error 40 us; skew 500 ppm
Sep 26 14:44:58 dist-test-slave-s7k6 krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:58.012780 4435 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:44:58.013532 4435 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:58.013598 4435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:58.013772 4435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:58.015937 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "83437ec308974e5d8f1a55d1d9c0c6ba"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "25e4ef3bfbb5b68d4cc5a4c59ca9f6bc"
server_key_iv: "897b94222e33d5aa50261f6f8b11166a"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.016469 4435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "83437ec308974e5d8f1a55d1d9c0c6ba"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "25e4ef3bfbb5b68d4cc5a4c59ca9f6bc"
server_key_iv: "897b94222e33d5aa50261f6f8b11166a"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.020479 4435 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:44:58.023029 4449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:58.024350 4435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:58.024472 4435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "83437ec308974e5d8f1a55d1d9c0c6ba"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "25e4ef3bfbb5b68d4cc5a4c59ca9f6bc"
server_key_iv: "897b94222e33d5aa50261f6f8b11166a"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.024585 4435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:58.050518 4435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:58.053823 4435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:58.053987 4435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:58.061183 4435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41195
I20260926 14:44:58.061194 4501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41195 every 8 connection(s)
I20260926 14:44:58.061882 4435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:44:58.065037 4502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:58.066126 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 4435
I20260926 14:44:58.066248 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:44:58.066570 392 external_mini_cluster.cc:1428] Setting key 0fcec511d19f9ca766ef8eefb683dc96
I20260926 14:44:58.071880 4502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba: Bootstrap starting.
I20260926 14:44:58.074425 4502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:58.075244 4502 log.cc:824] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:44:58 dist-test-slave-s7k6 krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:58.077435 4502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba: No bootstrap required, opened a new log
I20260926 14:44:58.081182 4502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83437ec308974e5d8f1a55d1d9c0c6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } }
I20260926 14:44:58.081430 4502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:58.081447 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.068293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.0.98.62:41195)
0926 14:44:58.068799 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:58.068816 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:58.068855 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:44:58.069518 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:58.069535 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:58.069921 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:58.070271 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:58.071112 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.072201 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.073087 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.073601 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.073764 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.076557 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:58.076563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:58.076578 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:58.076609 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:58.078341 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:58.078908 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:58.078913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:58.078918 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:58.079056 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:58.079344 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:58.079347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:58.079348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:58.079791 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:58.079996 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:58.080320 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 14:44:58.080645 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":141,"threads_started":1}
I20260926 14:44:58.081496 4502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83437ec308974e5d8f1a55d1d9c0c6ba, State: Initialized, Role: FOLLOWER
I20260926 14:44:58.082031 4502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83437ec308974e5d8f1a55d1d9c0c6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } }
I20260926 14:44:58.082195 4502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:44:58.082255 4502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:44:58.082350 4502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:58.083384 4502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83437ec308974e5d8f1a55d1d9c0c6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } }
I20260926 14:44:58.083762 4502 leader_election.cc:302] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83437ec308974e5d8f1a55d1d9c0c6ba; no voters:
I20260926 14:44:58.084077 4502 leader_election.cc:288] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:44:58.084177 4507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:58.084408 4507 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [term 1 LEADER]: Becoming Leader. State: Replica: 83437ec308974e5d8f1a55d1d9c0c6ba, State: Running, Role: LEADER
I20260926 14:44:58.084721 4507 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83437ec308974e5d8f1a55d1d9c0c6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } }
I20260926 14:44:58.085245 4502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:44:58.086541 4508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83437ec308974e5d8f1a55d1d9c0c6ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83437ec308974e5d8f1a55d1d9c0c6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } } }
I20260926 14:44:58.086567 4509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83437ec308974e5d8f1a55d1d9c0c6ba. Latest consensus state: current_term: 1 leader_uuid: "83437ec308974e5d8f1a55d1d9c0c6ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83437ec308974e5d8f1a55d1d9c0c6ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41195 } } }
I20260926 14:44:58.086730 4509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:58.087015 4508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:58.087203 4518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:44:58.090365 4518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:44:58.096295 4518 catalog_manager.cc:1347] Generated new cluster ID: f54668ad0fac494cb1e50cd3d29594ee
I20260926 14:44:58.096367 4518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:44:58.108592 4518 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:44:58.109910 4518 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:44:58.120900 4518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba: Generated new TSK 0
I20260926 14:44:58.121691 4518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:44:58.157891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:58.275009 4528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:58.275302 4528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:58.275368 4528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:58.279676 4528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:58.279757 4528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:58.279839 4528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:44:58.286139 4528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:58.287398 4528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:58.288491 4528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:58.298349 4534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:58.298367 4536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:58.298341 4533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:58.298628 4528 server_base.cc:1034] running on GCE node
I20260926 14:44:58.299175 4528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:58.299870 4528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:58.301060 4528 hybrid_clock.cc:648] HybridClock initialized: now 1790433898301047 us; error 38 us; skew 500 ppm
Sep 26 14:44:58 dist-test-slave-s7k6 krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:58.304133 4528 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:44:58.304905 4528 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:58.304982 4528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:58.305159 4528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:58.307372 4528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "3a2f2cfff06c4cc2a6b15148d72844fb"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "f329570a20c8a8acae4470a88333a73f"
server_key_iv: "50b50deca8d175f006d22b347044ff70"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.307950 4528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3a2f2cfff06c4cc2a6b15148d72844fb"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "f329570a20c8a8acae4470a88333a73f"
server_key_iv: "50b50deca8d175f006d22b347044ff70"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.311936 4528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:44:58.314800 4542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:58.316303 4528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:44:58.316425 4528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "3a2f2cfff06c4cc2a6b15148d72844fb"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "f329570a20c8a8acae4470a88333a73f"
server_key_iv: "50b50deca8d175f006d22b347044ff70"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.316560 4528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:58.332983 4528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:58.336303 4528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:58.336480 4528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:58.337082 4528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:58.337956 4528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:58.338006 4528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:58.338049 4528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:58.338078 4528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:58.346851 4528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42387
I20260926 14:44:58.346865 4655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42387 every 8 connection(s)
I20260926 14:44:58.347466 4528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:44:58.354948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 4528
I20260926 14:44:58.355086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:44:58.355360 392 external_mini_cluster.cc:1428] Setting key d9037d200ae28286846e5a82a9198d15
Sep 26 14:44:58 dist-test-slave-s7k6 krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:58.362902 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.349272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47669 (local address 127.0.98.62:41195)
0926 14:44:58.349420 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:58.349424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:58.350349 (+ 925us) server_negotiation.cc:407] Connection header received
0926 14:44:58.351842 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:58.351846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:58.351902 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:58.351982 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:58.353329 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.353943 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.355404 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.355813 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.355915 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.358921 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:58.358925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:58.358927 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:58.358953 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:58.360411 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:58.361103 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:58.361106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:58.361107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:58.361190 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:58.361624 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:58.361626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:58.361627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:58.361808 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:58.361879 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:58.362616 (+ 737us) server_negotiation.cc:299] Negotiation successful
0926 14:44:58.362730 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:44:58.364109 4658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.349691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41195 (local address 127.0.98.1:47669)
0926 14:44:58.350180 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:58.350227 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:58.351593 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:58.352187 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:58.352195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:58.352594 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:58.353154 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.353171 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.354108 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.354111 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.355259 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.355273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.356022 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.356027 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.356221 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.356911 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:58.356935 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:58.358706 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:58.360576 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:58.360583 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:58.360602 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:58.360987 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:58.361277 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:58.361280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:58.361283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:58.361483 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:58.361906 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:58.361914 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:58.362278 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 14:44:58.362522 (+ 244us) client_negotiation.cc:240] Negotiation successful
0926 14:44:58.362950 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 14:44:58.365541 4656 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41195
I20260926 14:44:58.365809 4656 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:58.366271 4656 heartbeater.cc:511] Master 127.0.98.62:41195 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 14:44:58.367849 4466 ts_manager.cc:194] Registered new tserver with Master: 3a2f2cfff06c4cc2a6b15148d72844fb (127.0.98.1:42387)
I20260926 14:44:58.369329 4466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:44:58.382292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:58.496692 4661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:58.496994 4661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:58.497061 4661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:58.501456 4661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:58.501543 4661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:58.501631 4661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:44:58.507222 4661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:58.508353 4661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:58.509431 4661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:58.519630 4666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:58.519634 4669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:58.519924 4661 server_base.cc:1034] running on GCE node
W20260926 14:44:58.519630 4667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:58.520579 4661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:58.521369 4661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:58.522543 4661 hybrid_clock.cc:648] HybridClock initialized: now 1790433898522514 us; error 45 us; skew 500 ppm
Sep 26 14:44:58 dist-test-slave-s7k6 krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:58.525486 4661 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:44:58.526310 4661 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:58.526381 4661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:58.526583 4661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:58.528824 4661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "cac19dc5ccd74dbfb1bc272460332986"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "cfbd95f3743d1d0bd9487da729750dc9"
server_key_iv: "4748966549f189cef9b1afac6272e958"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.529392 4661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cac19dc5ccd74dbfb1bc272460332986"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "cfbd95f3743d1d0bd9487da729750dc9"
server_key_iv: "4748966549f189cef9b1afac6272e958"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.533332 4661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 14:44:58.535934 4675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:58.537307 4661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:44:58.537431 4661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "cac19dc5ccd74dbfb1bc272460332986"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "cfbd95f3743d1d0bd9487da729750dc9"
server_key_iv: "4748966549f189cef9b1afac6272e958"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.537540 4661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:58.551510 4661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:58.554519 4661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:58.554708 4661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:58.555320 4661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:58.556193 4661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:58.556242 4661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:58.556298 4661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:58.556326 4661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:58.564815 4661 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46525
I20260926 14:44:58.564836 4788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46525 every 8 connection(s)
I20260926 14:44:58.565456 4661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:44:58.568674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 4661
I20260926 14:44:58.568818 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:44:58.569232 392 external_mini_cluster.cc:1428] Setting key e597bfd95e173721f362578d035f27e3
Sep 26 14:44:58 dist-test-slave-s7k6 krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:44:58.582055 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.567286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35021 (local address 127.0.98.62:41195)
0926 14:44:58.567423 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:58.567427 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:58.568344 (+ 917us) server_negotiation.cc:407] Connection header received
0926 14:44:58.570081 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:58.570085 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:58.570140 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:58.570219 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:58.571908 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.572451 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.574220 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.574459 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.574618 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.578062 (+ 3444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:58.578066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:58.578068 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:58.578096 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:58.579679 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:58.580292 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:58.580295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:58.580297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:58.580386 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:58.580864 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:58.580868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:58.580871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:58.581087 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:58.581183 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:58.581771 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 14:44:58.581893 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:44:58.582976 4791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.567704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41195 (local address 127.0.98.2:35021)
0926 14:44:58.568180 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:58.568227 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:58.569849 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:58.570417 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:58.570425 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:58.570980 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:58.571722 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.571744 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.572638 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.572642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.574053 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.574067 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.574710 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.574715 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.574979 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.575812 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:58.575841 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:58.577838 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:58.579804 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:58.579810 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:58.579823 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:58.580174 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:58.580541 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:58.580545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:58.580547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:58.580735 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:58.581196 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:58.581204 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:58.581477 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 14:44:58.581679 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 14:44:58.582045 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260926 14:44:58.584434 4789 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41195
I20260926 14:44:58.584720 4789 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:58.585338 4789 heartbeater.cc:511] Master 127.0.98.62:41195 requested a full tablet report, sending...
I20260926 14:44:58.586679 4466 ts_manager.cc:194] Registered new tserver with Master: cac19dc5ccd74dbfb1bc272460332986 (127.0.98.2:46525)
I20260926 14:44:58.587369 4466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35021
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:44:58.598127 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:44:58.711923 4794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:58.712234 4794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:58.712301 4794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:58.716696 4794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:58.716766 4794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:58.716851 4794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:44:58.722952 4794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:58.724399 4794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:58.725642 4794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:58.735246 4799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:58.735258 4800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:58.735246 4802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:58.735350 4794 server_base.cc:1034] running on GCE node
I20260926 14:44:58.735992 4794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:58.736670 4794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:58.737838 4794 hybrid_clock.cc:648] HybridClock initialized: now 1790433898737812 us; error 42 us; skew 500 ppm
Sep 26 14:44:58 dist-test-slave-s7k6 krb5kdc[4422](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:44:58.740859 4794 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:44:58.741637 4794 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:58.741708 4794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:58.741905 4794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:58.744009 4794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "a52711907aa74dbd9820d593367c46b0"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "ed2bfe039fb7538de6996018ef7e4380"
server_key_iv: "345da89f9443edafa9c4c205e7d4b87d"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.744561 4794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a52711907aa74dbd9820d593367c46b0"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "ed2bfe039fb7538de6996018ef7e4380"
server_key_iv: "345da89f9443edafa9c4c205e7d4b87d"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.748423 4794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:44:58.751124 4808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:58.752599 4794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:44:58.752770 4794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "a52711907aa74dbd9820d593367c46b0"
format_stamp: "Formatted at 2026-09-26 14:44:58 on dist-test-slave-s7k6"
server_key: "ed2bfe039fb7538de6996018ef7e4380"
server_key_iv: "345da89f9443edafa9c4c205e7d4b87d"
server_key_version: "encryptionkey@0"
I20260926 14:44:58.752899 4794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:58.769306 4794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:58.772647 4794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:58.772820 4794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:58.773393 4794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:58.774201 4794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:58.774247 4794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:58.774298 4794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:58.774325 4794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:58.782514 4794 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35749
I20260926 14:44:58.782547 4921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35749 every 8 connection(s)
I20260926 14:44:58.783175 4794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:44:58.785161 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 4794
I20260926 14:44:58.785280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:44:58.785545 392 external_mini_cluster.cc:1428] Setting key c701d429b59d79a7ccb34a32c55469aa
Sep 26 14:44:58 dist-test-slave-s7k6 krb5kdc[4422](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:44:58.796865 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.785049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33817 (local address 127.0.98.62:41195)
0926 14:44:58.785197 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:58.785202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:58.786276 (+ 1074us) server_negotiation.cc:407] Connection header received
0926 14:44:58.787771 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:58.787775 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:58.787831 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:58.787903 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:58.789099 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.789623 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.790848 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.791090 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.791181 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.793749 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:58.793752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:58.793755 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:58.793783 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:58.795057 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:58.795534 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:58.795536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:58.795538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:58.795605 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:58.795880 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:58.795882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:58.795883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:58.796060 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:58.796131 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:58.796603 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 14:44:58.796737 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:44:58.797693 4924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.785575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41195 (local address 127.0.98.3:33817)
0926 14:44:58.786117 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:58.786183 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:58.787563 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:58.788082 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:58.788090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:58.788467 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:58.788961 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.788975 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.789747 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.789751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.790694 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.790704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.791239 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.791243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.791415 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.791993 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:44:58.792010 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:58.793551 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:58.795171 (+ 1620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:58.795176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:58.795191 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:58.795454 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:58.795670 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:44:58.795673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:44:58.795674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:44:58.795811 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:44:58.796133 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:58.796139 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:58.796390 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 14:44:58.796551 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 14:44:58.796831 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":111,"threads_started":1}
I20260926 14:44:58.799003 4922 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41195
I20260926 14:44:58.799263 4922 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:58.799744 4922 heartbeater.cc:511] Master 127.0.98.62:41195 requested a full tablet report, sending...
I20260926 14:44:58.800774 4466 ts_manager.cc:194] Registered new tserver with Master: a52711907aa74dbd9820d593367c46b0 (127.0.98.3:35749)
I20260926 14:44:58.801354 4466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33817
I20260926 14:44:58.811767 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:44:58.821065 4505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.813760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.0.98.62:41195)
0926 14:44:58.813885 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:58.813890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:58.814118 (+ 228us) server_negotiation.cc:407] Connection header received
0926 14:44:58.814261 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:58.814264 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:58.814307 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:58.814368 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:58.815108 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.815637 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.816478 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.816695 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.816781 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.818135 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:58.818139 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:58.818141 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:44:58.818167 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:58.819478 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:58.819886 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:58.819890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:58.819891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:58.819966 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:44:58.820202 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:44:58.820206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:44:58.820208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:44:58.820428 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:58.820563 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:58.820801 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 14:44:58.820911 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:44:58.824026 4466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43126:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:44:58.827132 4466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:44:58.842000 4933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.836121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42387 (local address 127.0.0.1:60300)
0926 14:44:58.837168 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:58.837225 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:58.837358 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:58.837873 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:58.837879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:58.837900 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:58.838033 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.838040 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.839194 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.839197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.840411 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.840421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.841437 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.841441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.841546 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.841583 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 14:44:58.841727 (+ 144us) client_negotiation.cc:240] Negotiation successful
0926 14:44:58.841847 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"spinlock_wait_cycles":11008,"thread_start_us":92,"threads_started":1}
I20260926 14:44:58.842020 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.836245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.0.98.1:42387)
0926 14:44:58.837126 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:58.837134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:58.837411 (+ 277us) server_negotiation.cc:407] Connection header received
0926 14:44:58.837494 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:58.837500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:58.837713 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:58.837863 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:58.838145 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.839070 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.840554 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.841332 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.841426 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.841689 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:58.841754 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 14:44:58.841843 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":76,"threads_started":1}
I20260926 14:44:58.842515 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.836898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46525 (local address 127.0.0.1:53358)
0926 14:44:58.837749 (+ 851us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:58.837759 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:58.837836 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:58.838324 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:58.838327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:58.838344 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:58.838433 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.838438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.839755 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.839758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.841017 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.841027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.842160 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.842163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.842242 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.842262 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:44:58.842320 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:44:58.842381 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":4,"thread_start_us":83,"threads_started":1}
I20260926 14:44:58.842846 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.836526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.0.98.3:35749)
0926 14:44:58.837320 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:58.837331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:58.837355 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:44:58.837520 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:58.837525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:58.837713 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:58.837863 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:58.838272 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.839455 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.840872 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.841565 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.841821 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.841973 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:58.842573 (+ 600us) server_negotiation.cc:299] Negotiation successful
0926 14:44:58.842674 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":146,"threads_started":1}
I20260926 14:44:58.843007 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.837073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53358 (local address 127.0.98.2:46525)
0926 14:44:58.837635 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:58.837644 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:58.837917 (+ 273us) server_negotiation.cc:407] Connection header received
0926 14:44:58.838002 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:58.838007 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:58.838202 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:58.838354 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:58.838548 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.839622 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.841138 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.841890 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.841984 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.842101 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:58.842743 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 14:44:58.842869 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":106,"threads_started":1}
I20260926 14:44:58.845232 4590 tablet_service.cc:1467] Processing CreateTablet for tablet a7e8a98b1f03404fa996c7183cd41cf3 (DEFAULT_TABLE table=test-table [id=e646b9e2ed3e4a5db041a12c8f365b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:58.845427 4723 tablet_service.cc:1467] Processing CreateTablet for tablet a7e8a98b1f03404fa996c7183cd41cf3 (DEFAULT_TABLE table=test-table [id=e646b9e2ed3e4a5db041a12c8f365b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:58.845818 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.836213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35749 (local address 127.0.0.1:53618)
0926 14:44:58.837172 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:58.837227 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:58.837358 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:58.838034 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:58.838036 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:58.838050 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:58.838140 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.838145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.839587 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.839590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.840762 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.840770 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.841730 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.841734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.842432 (+ 698us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.842454 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:44:58.842520 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:44:58.842748 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":898,"mutex_wait_us":62,"spinlock_wait_cycles":2432,"thread_start_us":99,"threads_started":1}
I20260926 14:44:58.846334 4590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e8a98b1f03404fa996c7183cd41cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:58.846344 4723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e8a98b1f03404fa996c7183cd41cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:58.848743 4856 tablet_service.cc:1467] Processing CreateTablet for tablet a7e8a98b1f03404fa996c7183cd41cf3 (DEFAULT_TABLE table=test-table [id=e646b9e2ed3e4a5db041a12c8f365b7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:58.849845 4856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7e8a98b1f03404fa996c7183cd41cf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:58.852979 4939 tablet_bootstrap.cc:492] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb: Bootstrap starting.
I20260926 14:44:58.855134 4939 tablet_bootstrap.cc:654] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:58.855144 4940 tablet_bootstrap.cc:492] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986: Bootstrap starting.
I20260926 14:44:58.856262 4939 log.cc:824] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:58.857034 4942 tablet_bootstrap.cc:492] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0: Bootstrap starting.
I20260926 14:44:58.857420 4940 tablet_bootstrap.cc:654] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:58.858413 4939 tablet_bootstrap.cc:492] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb: No bootstrap required, opened a new log
I20260926 14:44:58.858599 4939 ts_tablet_manager.cc:1397] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:44:58.858610 4940 log.cc:824] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:58.859678 4942 tablet_bootstrap.cc:654] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:58.860255 4940 tablet_bootstrap.cc:492] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986: No bootstrap required, opened a new log
I20260926 14:44:58.860395 4940 ts_tablet_manager.cc:1397] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:44:58.860893 4942 log.cc:824] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:58.862425 4939 raft_consensus.cc:348] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42387 } } peers { permanent_uuid: "a52711907aa74dbd9820d593367c46b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35749 } } peers { permanent_uuid: "cac19dc5ccd74dbfb1bc272460332986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46525 } }
I20260926 14:44:58.862713 4939 raft_consensus.cc:374] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:58.862793 4939 raft_consensus.cc:729] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a2f2cfff06c4cc2a6b15148d72844fb, State: Initialized, Role: FOLLOWER
I20260926 14:44:58.862928 4942 tablet_bootstrap.cc:492] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0: No bootstrap required, opened a new log
I20260926 14:44:58.863173 4942 ts_tablet_manager.cc:1397] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:44:58.863262 4939 consensus_queue.cc:260] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42387 } } peers { permanent_uuid: "a52711907aa74dbd9820d593367c46b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35749 } } peers { permanent_uuid: "cac19dc5ccd74dbfb1bc272460332986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46525 } }
I20260926 14:44:58.863338 4940 raft_consensus.cc:348] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42387 } } peers { permanent_uuid: "a52711907aa74dbd9820d593367c46b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35749 } } peers { permanent_uuid: "cac19dc5ccd74dbfb1bc272460332986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46525 } }
I20260926 14:44:58.863601 4940 raft_consensus.cc:374] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:58.863663 4940 raft_consensus.cc:729] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cac19dc5ccd74dbfb1bc272460332986, State: Initialized, Role: FOLLOWER
I20260926 14:44:58.864045 4656 heartbeater.cc:503] Master 127.0.98.62:41195 was elected leader, sending a full tablet report...
I20260926 14:44:58.864108 4940 consensus_queue.cc:260] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42387 } } peers { permanent_uuid: "a52711907aa74dbd9820d593367c46b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35749 } } peers { permanent_uuid: "cac19dc5ccd74dbfb1bc272460332986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46525 } }
I20260926 14:44:58.864264 4939 ts_tablet_manager.cc:1428] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260926 14:44:58.865033 4940 ts_tablet_manager.cc:1428] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:44:58.865347 4789 heartbeater.cc:503] Master 127.0.98.62:41195 was elected leader, sending a full tablet report...
I20260926 14:44:58.867878 4942 raft_consensus.cc:348] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42387 } } peers { permanent_uuid: "a52711907aa74dbd9820d593367c46b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35749 } } peers { permanent_uuid: "cac19dc5ccd74dbfb1bc272460332986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46525 } }
I20260926 14:44:58.868177 4942 raft_consensus.cc:374] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:58.868238 4942 raft_consensus.cc:729] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52711907aa74dbd9820d593367c46b0, State: Initialized, Role: FOLLOWER
I20260926 14:44:58.868675 4942 consensus_queue.cc:260] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42387 } } peers { permanent_uuid: "a52711907aa74dbd9820d593367c46b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35749 } } peers { permanent_uuid: "cac19dc5ccd74dbfb1bc272460332986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46525 } }
I20260926 14:44:58.869315 4922 heartbeater.cc:503] Master 127.0.98.62:41195 was elected leader, sending a full tablet report...
I20260926 14:44:58.869603 4942 ts_tablet_manager.cc:1428] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:44:58.964749 4945 raft_consensus.cc:482] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:44:58.964984 4945 raft_consensus.cc:504] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42387 } } peers { permanent_uuid: "a52711907aa74dbd9820d593367c46b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35749 } } peers { permanent_uuid: "cac19dc5ccd74dbfb1bc272460332986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46525 } }
I20260926 14:44:58.966166 4945 leader_election.cc:288] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a52711907aa74dbd9820d593367c46b0 (127.0.98.3:35749), cac19dc5ccd74dbfb1bc272460332986 (127.0.98.2:46525)
I20260926 14:44:58.970350 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.966524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36309 (local address 127.0.98.3:35749)
0926 14:44:58.966687 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:58.966691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:58.966776 (+ 85us) server_negotiation.cc:407] Connection header received
0926 14:44:58.966968 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:58.966972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:58.967029 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:58.967115 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:58.967488 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.968105 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.969382 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.969805 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.969891 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.969955 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:58.970151 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 14:44:58.970224 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:44:58.970346 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.966942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38769 (local address 127.0.98.2:46525)
0926 14:44:58.967070 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:58.967074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:58.967235 (+ 161us) server_negotiation.cc:407] Connection header received
0926 14:44:58.967373 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:58.967377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:58.967429 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:58.967501 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:44:58.967807 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.968336 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.969412 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.969825 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.969920 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.969978 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:58.970153 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 14:44:58.970223 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:44:58.970366 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.966427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35749 (local address 127.0.98.1:36309)
0926 14:44:58.966687 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:58.966702 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:58.966856 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:58.967175 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:58.967177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:58.967197 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:58.967371 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.967379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.968225 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.968228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.969250 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.969260 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.969925 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.969929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.970024 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.970046 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:44:58.970109 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:44:58.970206 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":64,"spinlock_wait_cycles":19456,"thread_start_us":83,"threads_started":1}
I20260926 14:44:58.970374 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.966857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46525 (local address 127.0.98.1:38769)
0926 14:44:58.967135 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:58.967150 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:58.967274 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:58.967516 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:58.967518 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:58.967534 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:44:58.967701 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.967710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.968440 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.968443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.969279 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:44:58.969287 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.969943 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.969947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:44:58.970039 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.970057 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:44:58.970125 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:44:58.970229 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":50,"spinlock_wait_cycles":768,"thread_start_us":118,"threads_started":1}
I20260926 14:44:58.971024 4876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e8a98b1f03404fa996c7183cd41cf3" candidate_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a52711907aa74dbd9820d593367c46b0" is_pre_election: true
I20260926 14:44:58.971091 4743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e8a98b1f03404fa996c7183cd41cf3" candidate_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cac19dc5ccd74dbfb1bc272460332986" is_pre_election: true
I20260926 14:44:58.971405 4876 raft_consensus.cc:2454] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2f2cfff06c4cc2a6b15148d72844fb in term 0.
I20260926 14:44:58.971403 4743 raft_consensus.cc:2454] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a2f2cfff06c4cc2a6b15148d72844fb in term 0.
I20260926 14:44:58.972020 4543 leader_election.cc:302] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2f2cfff06c4cc2a6b15148d72844fb, a52711907aa74dbd9820d593367c46b0; no voters:
I20260926 14:44:58.972313 4945 raft_consensus.cc:2781] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:44:58.972404 4945 raft_consensus.cc:482] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:44:58.972452 4945 raft_consensus.cc:3037] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:58.973488 4945 raft_consensus.cc:504] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42387 } } peers { permanent_uuid: "a52711907aa74dbd9820d593367c46b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35749 } } peers { permanent_uuid: "cac19dc5ccd74dbfb1bc272460332986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46525 } }
I20260926 14:44:58.973817 4945 leader_election.cc:288] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [CANDIDATE]: Term 1 election: Requested vote from peers a52711907aa74dbd9820d593367c46b0 (127.0.98.3:35749), cac19dc5ccd74dbfb1bc272460332986 (127.0.98.2:46525)
I20260926 14:44:58.974179 4876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e8a98b1f03404fa996c7183cd41cf3" candidate_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a52711907aa74dbd9820d593367c46b0"
I20260926 14:44:58.974262 4743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e8a98b1f03404fa996c7183cd41cf3" candidate_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cac19dc5ccd74dbfb1bc272460332986"
I20260926 14:44:58.974298 4876 raft_consensus.cc:3037] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:58.974393 4743 raft_consensus.cc:3037] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:58.975394 4876 raft_consensus.cc:2454] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2f2cfff06c4cc2a6b15148d72844fb in term 1.
I20260926 14:44:58.975450 4743 raft_consensus.cc:2454] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a2f2cfff06c4cc2a6b15148d72844fb in term 1.
I20260926 14:44:58.975694 4543 leader_election.cc:302] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a2f2cfff06c4cc2a6b15148d72844fb, a52711907aa74dbd9820d593367c46b0; no voters:
I20260926 14:44:58.975863 4945 raft_consensus.cc:2781] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:58.976092 4945 raft_consensus.cc:686] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [term 1 LEADER]: Becoming Leader. State: Replica: 3a2f2cfff06c4cc2a6b15148d72844fb, State: Running, Role: LEADER
I20260926 14:44:58.976413 4945 consensus_queue.cc:237] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42387 } } peers { permanent_uuid: "a52711907aa74dbd9820d593367c46b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35749 } } peers { permanent_uuid: "cac19dc5ccd74dbfb1bc272460332986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46525 } }
I20260926 14:44:58.979741 4465 catalog_manager.cc:5495] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a2f2cfff06c4cc2a6b15148d72844fb (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a2f2cfff06c4cc2a6b15148d72844fb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42387 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a52711907aa74dbd9820d593367c46b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cac19dc5ccd74dbfb1bc272460332986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46525 } health_report { overall_health: UNKNOWN } } }
I20260926 14:44:58.994426 4935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:58.990911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.0.98.1:42387)
0926 14:44:58.991026 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:58.991029 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:58.991042 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:44:58.991132 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:58.991134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:58.991194 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:58.991286 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:44:58.991504 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.992027 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.992948 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:44:58.993184 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:44:58.993271 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:44:58.993568 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:44:58.994044 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:44:58.994176 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:58.994244 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 14:44:58.994302 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:44:59.000478 4876 raft_consensus.cc:1261] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0 [term 1 FOLLOWER]: Refusing update from remote peer 3a2f2cfff06c4cc2a6b15148d72844fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:59.000471 4743 raft_consensus.cc:1261] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986 [term 1 FOLLOWER]: Refusing update from remote peer 3a2f2cfff06c4cc2a6b15148d72844fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:44:59.001189 4945 consensus_queue.cc:1035] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cac19dc5ccd74dbfb1bc272460332986" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:59.001392 4950 consensus_queue.cc:1035] T a7e8a98b1f03404fa996c7183cd41cf3 P 3a2f2cfff06c4cc2a6b15148d72844fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "a52711907aa74dbd9820d593367c46b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:44:59.019593 4466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:59.019804 4466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43126:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:44:59.022541 4466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83437ec308974e5d8f1a55d1d9c0c6ba: Sending DeleteTablet for 3 replicas of tablet a7e8a98b1f03404fa996c7183cd41cf3
I20260926 14:44:59.023375 4590 tablet_service.cc:1514] Processing DeleteTablet for tablet a7e8a98b1f03404fa996c7183cd41cf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60300
I20260926 14:44:59.023452 4856 tablet_service.cc:1514] Processing DeleteTablet for tablet a7e8a98b1f03404fa996c7183cd41cf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53618
I20260926 14:44:59.023921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 4528
I20260926 14:44:59.024083 4723 tablet_service.cc:1514] Processing DeleteTablet for tablet a7e8a98b1f03404fa996c7183cd41cf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:44:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53358
I20260926 14:44:59.025466 4966 tablet_replica.cc:330] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986: stopping tablet replica
I20260926 14:44:59.025741 4965 tablet_replica.cc:330] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0: stopping tablet replica
I20260926 14:44:59.025843 4966 raft_consensus.cc:2229] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:59.026052 4965 raft_consensus.cc:2229] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:44:59.026172 4966 raft_consensus.cc:2258] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:59.026357 4965 raft_consensus.cc:2258] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:44:59.027675 4966 ts_tablet_manager.cc:1905] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:59.027889 4965 ts_tablet_manager.cc:1905] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:44:59.030041 4966 ts_tablet_manager.cc:1918] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:44:59.030138 4966 log.cc:1196] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/a7e8a98b1f03404fa996c7183cd41cf3
I20260926 14:44:59.030428 4966 ts_tablet_manager.cc:1939] T a7e8a98b1f03404fa996c7183cd41cf3 P cac19dc5ccd74dbfb1bc272460332986: Deleting consensus metadata
I20260926 14:44:59.030616 4965 ts_tablet_manager.cc:1918] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:44:59.030711 4965 log.cc:1196] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/a7e8a98b1f03404fa996c7183cd41cf3
I20260926 14:44:59.031024 4965 ts_tablet_manager.cc:1939] T a7e8a98b1f03404fa996c7183cd41cf3 P a52711907aa74dbd9820d593367c46b0: Deleting consensus metadata
W20260926 14:44:59.031319 4452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42387 (error 108)
I20260926 14:44:59.031543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 4661
I20260926 14:44:59.033085 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.032798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42387 (local address 127.0.0.1:60312)
0926 14:44:59.032933 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:59.033001 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 14:44:59.033253 4452 catalog_manager.cc:4584] TS 3a2f2cfff06c4cc2a6b15148d72844fb (127.0.98.1:42387): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7e8a98b1f03404fa996c7183cd41cf3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42387: connect: Connection refused (error 111)
I20260926 14:44:59.033391 4450 catalog_manager.cc:4841] TS a52711907aa74dbd9820d593367c46b0 (127.0.98.3:35749): tablet a7e8a98b1f03404fa996c7183cd41cf3 (table test-table [id=e646b9e2ed3e4a5db041a12c8f365b7a]) successfully deleted
I20260926 14:44:59.037317 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 4794
I20260926 14:44:59.038137 4934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.037524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46525 (local address 127.0.0.1:53370)
0926 14:44:59.037992 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:59.038049 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388}
I20260926 14:44:59.042605 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 4435
2026-09-26T14:44:59Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2242 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T14:44:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:44:59Z Disabled control of system clock
I20260926 14:44:59.070643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44459
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 14:44:59.183693 4971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:59.184003 4971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:59.184038 4971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:59.188199 4971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:44:59.188278 4971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:59.188292 4971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:59.188308 4971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:44:59.188321 4971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:44:59.193241 4971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44459
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:59.194365 4971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:59.195887 4971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:59.204428 4979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:59.204481 4977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:59.204481 4976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:59.204759 4971 server_base.cc:1034] running on GCE node
I20260926 14:44:59.205336 4971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:59.206481 4971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:59.207643 4971 hybrid_clock.cc:648] HybridClock initialized: now 1790433899207612 us; error 44 us; skew 500 ppm
I20260926 14:44:59.209512 4971 webserver.cc:460] Webserver started at http://127.0.98.62:45085/ using document root <none> and password file <none>
I20260926 14:44:59.210094 4971 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:59.210165 4971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:59.210357 4971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:59.212460 4971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "81d9b39657c5488782445fa7c10d500a"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "ee25ced428e1b3724d641db7c560568f"
server_key_iv: "b17cf84ce82400d70bd8ce656144a6d1"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.213003 4971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "81d9b39657c5488782445fa7c10d500a"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "ee25ced428e1b3724d641db7c560568f"
server_key_iv: "b17cf84ce82400d70bd8ce656144a6d1"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.216826 4971 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 14:44:59.219350 4985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:59.220654 4971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:44:59.220836 4971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "81d9b39657c5488782445fa7c10d500a"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "ee25ced428e1b3724d641db7c560568f"
server_key_iv: "b17cf84ce82400d70bd8ce656144a6d1"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.220945 4971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:59.225708 4971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:59.226327 4971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:59.226483 4971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:59.233320 4971 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44459
I20260926 14:44:59.233325 5037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44459 every 8 connection(s)
I20260926 14:44:59.234299 4971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:44:59.236152 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 4971
I20260926 14:44:59.236286 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:44:59.236588 392 external_mini_cluster.cc:1428] Setting key c40fe4fe02cb9958674e379def4a7ca5
I20260926 14:44:59.238862 5038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:59.242800 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.237952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.0.98.62:44459)
0926 14:44:59.239054 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:59.239071 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:59.239110 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:44:59.239812 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:59.239830 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:59.240282 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:59.240647 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:59.241486 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:59.241491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:59.241504 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:44:59.241514 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:59.241572 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:44:59.241588 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:59.241722 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:59.241791 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 14:44:59.242035 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":519,"threads_started":1}
I20260926 14:44:59.244930 5038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a: Bootstrap starting.
I20260926 14:44:59.247169 5038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:59.247918 5038 log.cc:824] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:59.249951 5038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a: No bootstrap required, opened a new log
I20260926 14:44:59.253584 5038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9b39657c5488782445fa7c10d500a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260926 14:44:59.253823 5038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:59.253867 5038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81d9b39657c5488782445fa7c10d500a, State: Initialized, Role: FOLLOWER
I20260926 14:44:59.254307 5038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9b39657c5488782445fa7c10d500a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260926 14:44:59.254433 5038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:44:59.254493 5038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:44:59.254585 5038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:44:59.255548 5038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9b39657c5488782445fa7c10d500a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260926 14:44:59.255879 5038 leader_election.cc:302] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81d9b39657c5488782445fa7c10d500a; no voters:
I20260926 14:44:59.256171 5038 leader_election.cc:288] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:44:59.256268 5043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:44:59.256501 5043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [term 1 LEADER]: Becoming Leader. State: Replica: 81d9b39657c5488782445fa7c10d500a, State: Running, Role: LEADER
I20260926 14:44:59.256830 5043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9b39657c5488782445fa7c10d500a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } }
I20260926 14:44:59.257416 5038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:44:59.258666 5044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81d9b39657c5488782445fa7c10d500a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9b39657c5488782445fa7c10d500a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } } }
I20260926 14:44:59.258838 5044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:59.258666 5045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81d9b39657c5488782445fa7c10d500a. Latest consensus state: current_term: 1 leader_uuid: "81d9b39657c5488782445fa7c10d500a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81d9b39657c5488782445fa7c10d500a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44459 } } }
I20260926 14:44:59.259075 5045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a [sys.catalog]: This master's current role is: LEADER
I20260926 14:44:59.259339 5053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:44:59.262737 5053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:44:59.271538 5053 catalog_manager.cc:1347] Generated new cluster ID: c71a64675c3947729ac97a57d4940a02
I20260926 14:44:59.271628 5053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:44:59.291635 5053 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:44:59.291970 5053 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:44:59.299734 5053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a: Generated new TSK 0
I20260926 14:44:59.300498 5053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:44:59.313000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--builtin_ntp_servers=127.0.98.20:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 14:44:59.427034 5062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:59.427316 5062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:59.427372 5062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:59.431565 5062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:59.431631 5062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:59.431722 5062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:44:59.436769 5062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:59.437968 5062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:59.439523 5062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:59.449760 5070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:59.449973 5068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:59.450011 5067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:59.450124 5062 server_base.cc:1034] running on GCE node
I20260926 14:44:59.450647 5062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:59.451339 5062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:59.452531 5062 hybrid_clock.cc:648] HybridClock initialized: now 1790433899452509 us; error 35 us; skew 500 ppm
I20260926 14:44:59.454610 5062 webserver.cc:460] Webserver started at http://127.0.98.1:33403/ using document root <none> and password file <none>
I20260926 14:44:59.455299 5062 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:59.455370 5062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:59.455569 5062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:59.457799 5062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "7e111277a10047e783d63bd7e96d4d64"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "fa78af0c6d2a74d5afa5012261b4d5ed"
server_key_iv: "0db5f181d9d7ea61e08616f72a48eae6"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.458376 5062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7e111277a10047e783d63bd7e96d4d64"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "fa78af0c6d2a74d5afa5012261b4d5ed"
server_key_iv: "0db5f181d9d7ea61e08616f72a48eae6"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.462329 5062 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:44:59.465030 5076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:59.466326 5062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:44:59.466449 5062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "7e111277a10047e783d63bd7e96d4d64"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "fa78af0c6d2a74d5afa5012261b4d5ed"
server_key_iv: "0db5f181d9d7ea61e08616f72a48eae6"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.466550 5062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:59.471127 5062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:59.471755 5062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:59.471910 5062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:59.472445 5062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:59.473403 5062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:59.473448 5062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:59.473498 5062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:59.473522 5062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:59.482283 5062 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37753
I20260926 14:44:59.482316 5189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37753 every 8 connection(s)
I20260926 14:44:59.483235 5062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:44:59.489405 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.485095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53855 (local address 127.0.98.62:44459)
0926 14:44:59.485265 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:59.485269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:59.486358 (+ 1089us) server_negotiation.cc:407] Connection header received
0926 14:44:59.487695 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:59.487698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:59.487738 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:59.487818 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:59.488951 (+ 1133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:59.488954 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:59.488956 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:44:59.488968 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:59.488989 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:44:59.488997 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:59.489086 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:59.489178 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 14:44:59.489263 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 14:44:59.489539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 5062
I20260926 14:44:59.489647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:44:59.489948 392 external_mini_cluster.cc:1428] Setting key d052852647005eff858f2b084b9effc7
I20260926 14:44:59.490268 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.485492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.1:53855)
0926 14:44:59.486180 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:59.486238 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:59.487476 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:59.487977 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:59.487984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:59.488088 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:59.488665 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:44:59.488666 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:59.488711 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:44:59.488785 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:59.489074 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:59.489078 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:59.489082 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:44:59.489206 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 14:44:59.489422 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":134,"threads_started":1}
I20260926 14:44:59.491466 5190 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44459
I20260926 14:44:59.491685 5190 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:59.491899 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--builtin_ntp_servers=127.0.98.20:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 14:44:59.492117 5190 heartbeater.cc:511] Master 127.0.98.62:44459 requested a full tablet report, sending...
I20260926 14:44:59.493757 5002 ts_manager.cc:194] Registered new tserver with Master: 7e111277a10047e783d63bd7e96d4d64 (127.0.98.1:37753)
W20260926 14:44:59.603312 5193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:59.603596 5193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:59.603652 5193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:59.607779 5193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:59.607839 5193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:59.607916 5193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:44:59.612852 5193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:59.613986 5193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:59.615448 5193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:59.625456 5201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:59.625509 5198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:59.625465 5199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:59.625880 5193 server_base.cc:1034] running on GCE node
I20260926 14:44:59.626263 5193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:59.626936 5193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:59.628121 5193 hybrid_clock.cc:648] HybridClock initialized: now 1790433899628103 us; error 65 us; skew 500 ppm
I20260926 14:44:59.630162 5193 webserver.cc:460] Webserver started at http://127.0.98.2:39309/ using document root <none> and password file <none>
I20260926 14:44:59.630810 5193 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:59.630882 5193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:59.631080 5193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:59.633241 5193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "896d0c4eb24348bea40cc59403305df2"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "0cf175f67b20a9b7335fbd06217efa7e"
server_key_iv: "31e4f5a2d90bc75f69fee2d176ba165b"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.633823 5193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "896d0c4eb24348bea40cc59403305df2"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "0cf175f67b20a9b7335fbd06217efa7e"
server_key_iv: "31e4f5a2d90bc75f69fee2d176ba165b"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.638047 5193 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:44:59.640939 5207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:59.642377 5193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 14:44:59.642508 5193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "896d0c4eb24348bea40cc59403305df2"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "0cf175f67b20a9b7335fbd06217efa7e"
server_key_iv: "31e4f5a2d90bc75f69fee2d176ba165b"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.642622 5193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:59.647399 5193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:59.648030 5193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:59.648187 5193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:59.648736 5193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:59.649650 5193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:59.649696 5193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:59.649746 5193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:59.649775 5193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:59.658406 5193 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42923
I20260926 14:44:59.658453 5320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42923 every 8 connection(s)
I20260926 14:44:59.659389 5193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:44:59.665660 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.661277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40319 (local address 127.0.98.62:44459)
0926 14:44:59.661421 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:59.661424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:59.662557 (+ 1133us) server_negotiation.cc:407] Connection header received
0926 14:44:59.664004 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:59.664007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:59.664053 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:59.664128 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:59.665230 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:59.665233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:59.665235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:44:59.665246 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:59.665265 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:44:59.665273 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:59.665351 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:59.665478 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 14:44:59.665523 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:44:59.666532 5323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.661681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.2:40319)
0926 14:44:59.662397 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:59.662440 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:59.663776 (+ 1336us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:59.664319 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:59.664327 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:59.664424 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:59.664974 (+ 550us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:44:59.664975 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:59.665017 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:44:59.665088 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:59.665342 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:59.665351 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:59.665356 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:44:59.665518 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 14:44:59.665773 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":114,"threads_started":1}
I20260926 14:44:59.667744 5321 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44459
I20260926 14:44:59.667955 5321 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:59.668310 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 5193
I20260926 14:44:59.668432 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:44:59.668455 5321 heartbeater.cc:511] Master 127.0.98.62:44459 requested a full tablet report, sending...
I20260926 14:44:59.668766 392 external_mini_cluster.cc:1428] Setting key 26db5fdc510a839d1975972c0b54d054
I20260926 14:44:59.669783 5001 ts_manager.cc:194] Registered new tserver with Master: 896d0c4eb24348bea40cc59403305df2 (127.0.98.2:42923)
I20260926 14:44:59.671011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--builtin_ntp_servers=127.0.98.20:44499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 14:44:59.787575 5324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:44:59.787868 5324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:44:59.787926 5324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:44:59.792068 5324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:44:59.792140 5324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:44:59.792222 5324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:44:59.797341 5324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44459
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:44:59.798548 5324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:44:59.800274 5324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:44:59.810894 5330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:44:59.810883 5329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:59.811098 5324 server_base.cc:1034] running on GCE node
W20260926 14:44:59.810887 5332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:44:59.811848 5324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:44:59.812552 5324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:44:59.813789 5324 hybrid_clock.cc:648] HybridClock initialized: now 1790433899813721 us; error 43 us; skew 500 ppm
I20260926 14:44:59.815881 5324 webserver.cc:460] Webserver started at http://127.0.98.3:38733/ using document root <none> and password file <none>
I20260926 14:44:59.816496 5324 fs_manager.cc:359] Metadata directory not provided
I20260926 14:44:59.816566 5324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:44:59.816774 5324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:44:59.818969 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "b1b03866c2e5411e93785d1a61c266df"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "6dd5f1bca3d2465e3db50c5126bbe6e7"
server_key_iv: "81c4aca7c9902a67c9a0cff02697ff96"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.819525 5324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b1b03866c2e5411e93785d1a61c266df"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "6dd5f1bca3d2465e3db50c5126bbe6e7"
server_key_iv: "81c4aca7c9902a67c9a0cff02697ff96"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.823431 5324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:44:59.826200 5338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:59.827647 5324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:44:59.827765 5324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "b1b03866c2e5411e93785d1a61c266df"
format_stamp: "Formatted at 2026-09-26 14:44:59 on dist-test-slave-s7k6"
server_key: "6dd5f1bca3d2465e3db50c5126bbe6e7"
server_key_iv: "81c4aca7c9902a67c9a0cff02697ff96"
server_key_version: "encryptionkey@0"
I20260926 14:44:59.827874 5324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:44:59.832772 5324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:44:59.833419 5324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:44:59.833585 5324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:44:59.834139 5324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:44:59.835098 5324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:44:59.835144 5324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:59.835196 5324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:44:59.835223 5324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:44:59.843791 5324 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41753
I20260926 14:44:59.843811 5451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41753 every 8 connection(s)
I20260926 14:44:59.844774 5324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:44:59.847091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 5324
I20260926 14:44:59.847199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:44:59.847467 392 external_mini_cluster.cc:1428] Setting key 47ffdb9689f86c74179f267b0c91cccd
I20260926 14:44:59.851827 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.846644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35375 (local address 127.0.98.62:44459)
0926 14:44:59.846818 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:59.846823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:59.848480 (+ 1657us) server_negotiation.cc:407] Connection header received
0926 14:44:59.850085 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:59.850088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:59.850128 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:59.850202 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:59.851394 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:59.851397 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:59.851399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:44:59.851411 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:59.851432 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:44:59.851440 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:59.851525 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:59.851625 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 14:44:59.851663 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:44:59.852759 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.847222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44459 (local address 127.0.98.3:35375)
0926 14:44:59.848275 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:59.848331 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:59.849887 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:59.850399 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:59.850406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:59.850506 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:59.851085 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:44:59.851086 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:59.851133 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:44:59.851216 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:59.851515 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:59.851520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:59.851524 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:44:59.851656 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 14:44:59.851873 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":114,"threads_started":1}
I20260926 14:44:59.853919 5452 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44459
I20260926 14:44:59.854178 5452 heartbeater.cc:464] Registering TS with master...
I20260926 14:44:59.854592 5452 heartbeater.cc:511] Master 127.0.98.62:44459 requested a full tablet report, sending...
I20260926 14:44:59.855628 5001 ts_manager.cc:194] Registered new tserver with Master: b1b03866c2e5411e93785d1a61c266df (127.0.98.3:41753)
I20260926 14:44:59.862155 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:44:59.865767 5041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.864159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50294 (local address 127.0.98.62:44459)
0926 14:44:59.864307 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:59.864312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:59.864566 (+ 254us) server_negotiation.cc:407] Connection header received
0926 14:44:59.864640 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:59.864642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:59.864680 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:59.864747 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:59.865413 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:59.865416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:59.865418 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:44:59.865429 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:59.865449 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:44:59.865456 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:59.865530 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:59.865585 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 14:44:59.865618 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 14:44:59.868753 5001 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:44:59.871956 5001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:44:59.884362 5465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.881313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37753 (local address 127.0.0.1:57786)
0926 14:44:59.881633 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:59.881645 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:59.881708 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:59.883192 (+ 1484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:59.883195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:59.883210 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:59.883639 (+ 429us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:44:59.883640 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:59.883649 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:44:59.883668 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:59.883980 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:59.883989 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:59.883995 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:44:59.884108 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 14:44:59.884207 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":21,"thread_start_us":102,"threads_started":1}
I20260926 14:44:59.884390 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.881234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.0.98.2:42923)
0926 14:44:59.881938 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:59.881949 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:59.881974 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:44:59.882029 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:59.882034 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:59.882267 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:59.882406 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:59.883763 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:59.883770 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:59.883783 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:44:59.883797 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:59.883845 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:44:59.883881 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:59.883983 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:59.884134 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 14:44:59.884216 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":142,"threads_started":1}
I20260926 14:44:59.884509 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.881913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.0.98.1:37753)
0926 14:44:59.882507 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:59.882517 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:59.882539 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:44:59.882596 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:59.882602 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:59.883006 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:59.883175 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:59.883789 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:59.883795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:59.883806 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:44:59.883817 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:59.883856 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:44:59.883882 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:59.883982 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:59.884284 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 14:44:59.884350 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":125,"threads_started":1}
I20260926 14:44:59.884737 5463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.881064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42923 (local address 127.0.0.1:54786)
0926 14:44:59.881427 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:59.881483 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:59.881589 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:59.882829 (+ 1240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:59.882836 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:59.882923 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:59.883505 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:44:59.883507 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:59.883528 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:44:59.883602 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:59.884031 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:59.884033 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:59.884036 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:44:59.884108 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 14:44:59.884188 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":107,"threads_started":1}
I20260926 14:44:59.887419 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.881856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41753 (local address 127.0.0.1:50210)
0926 14:44:59.882536 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 14:44:59.882551 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:44:59.882680 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:44:59.883396 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:44:59.883399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:44:59.883412 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:44:59.884337 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:44:59.884339 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:44:59.884348 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:44:59.884371 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:44:59.887165 (+ 2794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:44:59.887169 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:44:59.887171 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 14:44:59.887252 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 14:44:59.887290 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":85,"threads_started":1}
I20260926 14:44:59.887552 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:44:59.882046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50210 (local address 127.0.98.3:41753)
0926 14:44:59.882394 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0926 14:44:59.882406 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:44:59.882625 (+ 219us) server_negotiation.cc:407] Connection header received
0926 14:44:59.882844 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:44:59.882851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:44:59.883002 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:44:59.883120 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:44:59.886862 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:44:59.886868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:44:59.886881 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:44:59.886898 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:44:59.886948 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:44:59.886987 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:44:59.887089 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:44:59.887307 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 14:44:59.887397 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":128,"threads_started":1}
I20260926 14:44:59.887311 5124 tablet_service.cc:1467] Processing CreateTablet for tablet a00155dbb5844506bdfd939aa32ce340 (DEFAULT_TABLE table=test-table [id=804a130356fc4101a8730f5bae8da7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:59.887322 5255 tablet_service.cc:1467] Processing CreateTablet for tablet a00155dbb5844506bdfd939aa32ce340 (DEFAULT_TABLE table=test-table [id=804a130356fc4101a8730f5bae8da7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:59.888412 5124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00155dbb5844506bdfd939aa32ce340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:59.888411 5255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00155dbb5844506bdfd939aa32ce340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:59.890439 5386 tablet_service.cc:1467] Processing CreateTablet for tablet a00155dbb5844506bdfd939aa32ce340 (DEFAULT_TABLE table=test-table [id=804a130356fc4101a8730f5bae8da7f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:44:59.891544 5386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a00155dbb5844506bdfd939aa32ce340. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:44:59.894740 5469 tablet_bootstrap.cc:492] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64: Bootstrap starting.
I20260926 14:44:59.896447 5470 tablet_bootstrap.cc:492] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: Bootstrap starting.
I20260926 14:44:59.897185 5469 tablet_bootstrap.cc:654] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:59.898062 5469 log.cc:824] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:59.898506 5471 tablet_bootstrap.cc:492] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: Bootstrap starting.
I20260926 14:44:59.898623 5470 tablet_bootstrap.cc:654] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:59.899490 5470 log.cc:824] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:59.900180 5469 tablet_bootstrap.cc:492] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64: No bootstrap required, opened a new log
I20260926 14:44:59.900372 5469 ts_tablet_manager.cc:1397] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260926 14:44:59.900555 5471 tablet_bootstrap.cc:654] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: Neither blocks nor log segments found. Creating new log.
I20260926 14:44:59.901213 5471 log.cc:824] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: Log is configured to *not* fsync() on all Append() calls
I20260926 14:44:59.901533 5470 tablet_bootstrap.cc:492] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: No bootstrap required, opened a new log
I20260926 14:44:59.901721 5470 ts_tablet_manager.cc:1397] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260926 14:44:59.902779 5471 tablet_bootstrap.cc:492] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: No bootstrap required, opened a new log
I20260926 14:44:59.902995 5471 ts_tablet_manager.cc:1397] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:44:59.904973 5469 raft_consensus.cc:348] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896d0c4eb24348bea40cc59403305df2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42923 } } peers { permanent_uuid: "b1b03866c2e5411e93785d1a61c266df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "7e111277a10047e783d63bd7e96d4d64" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } }
I20260926 14:44:59.905253 5469 raft_consensus.cc:374] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:59.905308 5469 raft_consensus.cc:729] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e111277a10047e783d63bd7e96d4d64, State: Initialized, Role: FOLLOWER
I20260926 14:44:59.905784 5469 consensus_queue.cc:260] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896d0c4eb24348bea40cc59403305df2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42923 } } peers { permanent_uuid: "b1b03866c2e5411e93785d1a61c266df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "7e111277a10047e783d63bd7e96d4d64" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } }
I20260926 14:44:59.906062 5470 raft_consensus.cc:348] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896d0c4eb24348bea40cc59403305df2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42923 } } peers { permanent_uuid: "b1b03866c2e5411e93785d1a61c266df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "7e111277a10047e783d63bd7e96d4d64" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } }
I20260926 14:44:59.906327 5470 raft_consensus.cc:374] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:59.906388 5470 raft_consensus.cc:729] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 896d0c4eb24348bea40cc59403305df2, State: Initialized, Role: FOLLOWER
I20260926 14:44:59.906644 5190 heartbeater.cc:503] Master 127.0.98.62:44459 was elected leader, sending a full tablet report...
I20260926 14:44:59.906746 5471 raft_consensus.cc:348] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896d0c4eb24348bea40cc59403305df2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42923 } } peers { permanent_uuid: "b1b03866c2e5411e93785d1a61c266df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "7e111277a10047e783d63bd7e96d4d64" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } }
I20260926 14:44:59.906927 5469 ts_tablet_manager.cc:1428] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260926 14:44:59.906983 5471 raft_consensus.cc:374] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:44:59.907032 5471 raft_consensus.cc:729] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b03866c2e5411e93785d1a61c266df, State: Initialized, Role: FOLLOWER
I20260926 14:44:59.907419 5471 consensus_queue.cc:260] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896d0c4eb24348bea40cc59403305df2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42923 } } peers { permanent_uuid: "b1b03866c2e5411e93785d1a61c266df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "7e111277a10047e783d63bd7e96d4d64" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } }
I20260926 14:44:59.907496 5470 consensus_queue.cc:260] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896d0c4eb24348bea40cc59403305df2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42923 } } peers { permanent_uuid: "b1b03866c2e5411e93785d1a61c266df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "7e111277a10047e783d63bd7e96d4d64" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } }
I20260926 14:44:59.908103 5452 heartbeater.cc:503] Master 127.0.98.62:44459 was elected leader, sending a full tablet report...
I20260926 14:44:59.908317 5471 ts_tablet_manager.cc:1428] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:44:59.908545 5470 ts_tablet_manager.cc:1428] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260926 14:44:59.908891 5321 heartbeater.cc:503] Master 127.0.98.62:44459 was elected leader, sending a full tablet report...
W20260926 14:44:59.910290 5322 tablet.cc:2369] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:44:59.984390 5191 tablet.cc:2369] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:45:00.059393 5477 raft_consensus.cc:482] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:45:00.059607 5477 raft_consensus.cc:504] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896d0c4eb24348bea40cc59403305df2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42923 } } peers { permanent_uuid: "b1b03866c2e5411e93785d1a61c266df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "7e111277a10047e783d63bd7e96d4d64" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } }
I20260926 14:45:00.060748 5477 leader_election.cc:288] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1b03866c2e5411e93785d1a61c266df (127.0.98.3:41753), 7e111277a10047e783d63bd7e96d4d64 (127.0.98.1:37753)
I20260926 14:45:00.062772 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:00.061148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57727 (local address 127.0.98.1:37753)
0926 14:45:00.061281 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:00.061286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:00.061385 (+ 99us) server_negotiation.cc:407] Connection header received
0926 14:45:00.061498 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:00.061502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:00.061550 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:00.061629 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:00.062378 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:00.062381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:00.062383 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:00.062395 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:00.062414 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:00.062422 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:00.062501 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:00.062565 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 14:45:00.062610 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:45:00.062815 5323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:00.061052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37753 (local address 127.0.98.2:57727)
0926 14:45:00.061283 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:00.061299 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:00.061401 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:00.061665 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:00.061668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:00.061683 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:00.062219 (+ 536us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:45:00.062221 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:00.062230 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:45:00.062258 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:00.062489 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:00.062492 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:00.062495 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:45:00.062568 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:45:00.062618 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260926 14:45:00.062927 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:00.061148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53539 (local address 127.0.98.3:41753)
0926 14:45:00.061281 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:00.061286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:00.061612 (+ 326us) server_negotiation.cc:407] Connection header received
0926 14:45:00.061709 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:00.061711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:00.061750 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:00.061826 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:00.062520 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:00.062524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:00.062527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:00.062543 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:00.062560 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:00.062568 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:00.062649 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:00.062735 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 14:45:00.062796 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:45:00.063038 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:00.061052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41753 (local address 127.0.98.2:53539)
0926 14:45:00.061499 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:00.061512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:00.061620 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:00.061850 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:00.061852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:00.061863 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:00.062351 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:45:00.062354 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:00.062365 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:45:00.062390 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:00.062659 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:00.062662 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:00.062665 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 14:45:00.062735 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 14:45:00.062811 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":50,"thread_start_us":118,"threads_started":1}
I20260926 14:45:00.063575 5406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00155dbb5844506bdfd939aa32ce340" candidate_uuid: "896d0c4eb24348bea40cc59403305df2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b03866c2e5411e93785d1a61c266df" is_pre_election: true
I20260926 14:45:00.063581 5144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00155dbb5844506bdfd939aa32ce340" candidate_uuid: "896d0c4eb24348bea40cc59403305df2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e111277a10047e783d63bd7e96d4d64" is_pre_election: true
I20260926 14:45:00.063859 5144 raft_consensus.cc:2454] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 896d0c4eb24348bea40cc59403305df2 in term 0.
I20260926 14:45:00.063896 5406 raft_consensus.cc:2454] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 896d0c4eb24348bea40cc59403305df2 in term 0.
I20260926 14:45:00.064335 5211 leader_election.cc:302] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 896d0c4eb24348bea40cc59403305df2, b1b03866c2e5411e93785d1a61c266df; no voters:
I20260926 14:45:00.064601 5477 raft_consensus.cc:2781] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:45:00.064689 5477 raft_consensus.cc:482] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:45:00.064736 5477 raft_consensus.cc:3037] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:00.065811 5477 raft_consensus.cc:504] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896d0c4eb24348bea40cc59403305df2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42923 } } peers { permanent_uuid: "b1b03866c2e5411e93785d1a61c266df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "7e111277a10047e783d63bd7e96d4d64" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } }
I20260926 14:45:00.066164 5477 leader_election.cc:288] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [CANDIDATE]: Term 1 election: Requested vote from peers b1b03866c2e5411e93785d1a61c266df (127.0.98.3:41753), 7e111277a10047e783d63bd7e96d4d64 (127.0.98.1:37753)
I20260926 14:45:00.066548 5406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00155dbb5844506bdfd939aa32ce340" candidate_uuid: "896d0c4eb24348bea40cc59403305df2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1b03866c2e5411e93785d1a61c266df"
I20260926 14:45:00.066576 5144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a00155dbb5844506bdfd939aa32ce340" candidate_uuid: "896d0c4eb24348bea40cc59403305df2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e111277a10047e783d63bd7e96d4d64"
I20260926 14:45:00.066692 5144 raft_consensus.cc:3037] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:00.066694 5406 raft_consensus.cc:3037] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:00.068084 5144 raft_consensus.cc:2454] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896d0c4eb24348bea40cc59403305df2 in term 1.
I20260926 14:45:00.068084 5406 raft_consensus.cc:2454] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 896d0c4eb24348bea40cc59403305df2 in term 1.
I20260926 14:45:00.068423 5210 leader_election.cc:302] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e111277a10047e783d63bd7e96d4d64, 896d0c4eb24348bea40cc59403305df2; no voters:
I20260926 14:45:00.068626 5477 raft_consensus.cc:2781] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:00.068904 5477 raft_consensus.cc:686] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 1 LEADER]: Becoming Leader. State: Replica: 896d0c4eb24348bea40cc59403305df2, State: Running, Role: LEADER
I20260926 14:45:00.069326 5477 consensus_queue.cc:237] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896d0c4eb24348bea40cc59403305df2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42923 } } peers { permanent_uuid: "b1b03866c2e5411e93785d1a61c266df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } } peers { permanent_uuid: "7e111277a10047e783d63bd7e96d4d64" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } }
I20260926 14:45:00.073072 5001 catalog_manager.cc:5495] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 896d0c4eb24348bea40cc59403305df2 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "896d0c4eb24348bea40cc59403305df2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "896d0c4eb24348bea40cc59403305df2" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1b03866c2e5411e93785d1a61c266df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e111277a10047e783d63bd7e96d4d64" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 } health_report { overall_health: UNKNOWN } } }
W20260926 14:45:00.095769 5453 tablet.cc:2369] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:45:00.122792 5464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:00.121036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.0.98.2:42923)
0926 14:45:00.121155 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:00.121159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:00.121172 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:45:00.121387 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:00.121390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:00.121461 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:00.121547 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:00.122434 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:00.122438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:00.122440 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:00.122453 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:00.122473 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:00.122482 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:00.122565 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:00.122606 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 14:45:00.122644 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:45:00.129348 5144 raft_consensus.cc:1261] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64 [term 1 FOLLOWER]: Refusing update from remote peer 896d0c4eb24348bea40cc59403305df2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:00.129357 5406 raft_consensus.cc:1261] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df [term 1 FOLLOWER]: Refusing update from remote peer 896d0c4eb24348bea40cc59403305df2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:00.130115 5477 consensus_queue.cc:1035] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1b03866c2e5411e93785d1a61c266df" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:00.130359 5479 consensus_queue.cc:1035] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e111277a10047e783d63bd7e96d4d64" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:00.146610 5002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:00.146804 5002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:00.149161 5002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81d9b39657c5488782445fa7c10d500a: Sending DeleteTablet for 3 replicas of tablet a00155dbb5844506bdfd939aa32ce340
I20260926 14:45:00.150033 5124 tablet_service.cc:1514] Processing DeleteTablet for tablet a00155dbb5844506bdfd939aa32ce340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:00 UTC) from {username='slave'} at 127.0.0.1:57786
I20260926 14:45:00.150081 5255 tablet_service.cc:1514] Processing DeleteTablet for tablet a00155dbb5844506bdfd939aa32ce340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:00 UTC) from {username='slave'} at 127.0.0.1:54786
I20260926 14:45:00.150556 5491 tablet_replica.cc:330] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64: stopping tablet replica
I20260926 14:45:00.150578 5386 tablet_service.cc:1514] Processing DeleteTablet for tablet a00155dbb5844506bdfd939aa32ce340 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:00 UTC) from {username='slave'} at 127.0.0.1:50210
I20260926 14:45:00.151050 5491 raft_consensus.cc:2229] T a00155dbb5844506bdfd939aa32ce340 P 7e111277a10047e783d63bd7e96d4d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:00.151084 5492 tablet_replica.cc:330] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: stopping tablet replica
I20260926 14:45:00.151089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 5062
I20260926 14:45:00.152025 5492 raft_consensus.cc:2229] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:45:00.152468 5492 raft_consensus.cc:2258] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:00.153918 5492 ts_tablet_manager.cc:1905] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:00.156337 5492 ts_tablet_manager.cc:1918] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:00.156430 5492 log.cc:1196] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/a00155dbb5844506bdfd939aa32ce340
I20260926 14:45:00.156695 5492 ts_tablet_manager.cc:1939] T a00155dbb5844506bdfd939aa32ce340 P 896d0c4eb24348bea40cc59403305df2: Deleting consensus metadata
I20260926 14:45:00.157968 4989 catalog_manager.cc:4841] TS 896d0c4eb24348bea40cc59403305df2 (127.0.98.2:42923): tablet a00155dbb5844506bdfd939aa32ce340 (table test-table [id=804a130356fc4101a8730f5bae8da7f7]) successfully deleted
I20260926 14:45:00.159001 5493 tablet_replica.cc:330] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: stopping tablet replica
I20260926 14:45:00.159397 5493 raft_consensus.cc:2229] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:00.159767 5493 raft_consensus.cc:2258] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:45:00.159818 4988 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37753 (error 108)
I20260926 14:45:00.160017 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 5193
I20260926 14:45:00.162060 5466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:00.161784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37753 (local address 127.0.0.1:57802)
0926 14:45:00.161911 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:00.161984 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 14:45:00.162300 4988 catalog_manager.cc:4584] TS 7e111277a10047e783d63bd7e96d4d64 (127.0.98.1:37753): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a00155dbb5844506bdfd939aa32ce340: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37753: connect: Connection refused (error 111)
I20260926 14:45:00.162992 5493 ts_tablet_manager.cc:1905] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:00.165983 5493 ts_tablet_manager.cc:1918] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:00.166095 5493 log.cc:1196] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/a00155dbb5844506bdfd939aa32ce340
I20260926 14:45:00.166393 5493 ts_tablet_manager.cc:1939] T a00155dbb5844506bdfd939aa32ce340 P b1b03866c2e5411e93785d1a61c266df: Deleting consensus metadata
I20260926 14:45:00.166801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 5324
I20260926 14:45:00.171870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 4971
2026-09-26T14:45:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1129 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 14:45:00.192704 392 mini_oidc.cc:152] Starting JWKS server
I20260926 14:45:00.194370 392 webserver.cc:460] Webserver started at https://127.0.0.1:40671/ using document root <none> and password file <none>
I20260926 14:45:00.194523 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:45:00.194796 392 webserver.cc:460] Webserver started at http://127.0.0.1:34939/ using document root <none> and password file <none>
2026-09-26T14:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:00Z Disabled control of system clock
I20260926 14:45:00.203914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44317
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44317
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40671/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/testchainca.cert with env {}
W20260926 14:45:00.318276 5500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:00.318569 5500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:00.318626 5500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:00.322906 5500 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:00.322986 5500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:00.323007 5500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:00.323031 5500 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:00.323062 5500 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:45:00.323094 5500 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:45:00.323120 5500 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40671/jwks.json
I20260926 14:45:00.328928 5500 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44317
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40671/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:00.330070 5500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:00.331156 5500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:00.339576 5505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:00.339577 5508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:00.339730 5500 server_base.cc:1034] running on GCE node
W20260926 14:45:00.339635 5506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:00.340648 5500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:00.341830 5500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:00.342981 5500 hybrid_clock.cc:648] HybridClock initialized: now 1790433900342962 us; error 36 us; skew 500 ppm
I20260926 14:45:00.344889 5500 webserver.cc:460] Webserver started at http://127.0.98.62:46691/ using document root <none> and password file <none>
I20260926 14:45:00.345436 5500 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:00.345496 5500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:00.345654 5500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:00.347764 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "8920cf0c69004597b71d7602712ce823"
format_stamp: "Formatted at 2026-09-26 14:45:00 on dist-test-slave-s7k6"
server_key: "1133756f9d551c7c4cf9aa79e1839543"
server_key_iv: "36e06fe250aab7da8804a03f355377ef"
server_key_version: "encryptionkey@0"
I20260926 14:45:00.348296 5500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "8920cf0c69004597b71d7602712ce823"
format_stamp: "Formatted at 2026-09-26 14:45:00 on dist-test-slave-s7k6"
server_key: "1133756f9d551c7c4cf9aa79e1839543"
server_key_iv: "36e06fe250aab7da8804a03f355377ef"
server_key_version: "encryptionkey@0"
I20260926 14:45:00.352118 5500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:45:00.354697 5514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:00.356122 5500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:00.356261 5500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "8920cf0c69004597b71d7602712ce823"
format_stamp: "Formatted at 2026-09-26 14:45:00 on dist-test-slave-s7k6"
server_key: "1133756f9d551c7c4cf9aa79e1839543"
server_key_iv: "36e06fe250aab7da8804a03f355377ef"
server_key_version: "encryptionkey@0"
I20260926 14:45:00.356382 5500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:00.378986 5500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:00.379772 5500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:00.379968 5500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:00.386699 5500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44317
I20260926 14:45:00.386701 5566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44317 every 8 connection(s)
I20260926 14:45:00.387846 5500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:00.390234 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 5500
I20260926 14:45:00.390358 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:45:00.390694 392 external_mini_cluster.cc:1428] Setting key 3b195f45b77f365666d38053cba9bf69
I20260926 14:45:00.391538 5567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:00.398100 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823: Bootstrap starting.
I20260926 14:45:00.400557 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:00.392180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56632 (local address 127.0.98.62:44317)
0926 14:45:00.392853 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:00.392870 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:00.392911 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:45:00.393644 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:00.393664 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:00.394095 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:00.394428 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:00.394952 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.395987 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.396855 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.397347 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.397506 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:00.398418 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:00.398425 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:00.398442 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:00.398964 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:00.399037 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:00.399066 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:00.399171 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:00.399281 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 14:45:00.399585 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":151,"threads_started":1}
I20260926 14:45:00.401005 5567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:00.401902 5567 log.cc:824] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:00.403745 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:45:00.404090 5567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823: No bootstrap required, opened a new log
I20260926 14:45:00.407747 5567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8920cf0c69004597b71d7602712ce823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } }
I20260926 14:45:00.407996 5567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:00.408066 5567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8920cf0c69004597b71d7602712ce823, State: Initialized, Role: FOLLOWER
I20260926 14:45:00.408474 5567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8920cf0c69004597b71d7602712ce823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } }
I20260926 14:45:00.408600 5567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:00.408670 5567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:00.408761 5567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:00.409773 5567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8920cf0c69004597b71d7602712ce823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } }
I20260926 14:45:00.410091 5567 leader_election.cc:302] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8920cf0c69004597b71d7602712ce823; no voters:
I20260926 14:45:00.410394 5567 leader_election.cc:288] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:45:00.410501 5572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:00.410688 5572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [term 1 LEADER]: Becoming Leader. State: Replica: 8920cf0c69004597b71d7602712ce823, State: Running, Role: LEADER
I20260926 14:45:00.411037 5572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8920cf0c69004597b71d7602712ce823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } }
I20260926 14:45:00.411576 5567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:00.412647 5574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8920cf0c69004597b71d7602712ce823. Latest consensus state: current_term: 1 leader_uuid: "8920cf0c69004597b71d7602712ce823" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8920cf0c69004597b71d7602712ce823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } } }
I20260926 14:45:00.412667 5573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8920cf0c69004597b71d7602712ce823" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8920cf0c69004597b71d7602712ce823" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44317 } } }
I20260926 14:45:00.412786 5574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:00.412787 5573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:00.413143 5581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:00.415855 5581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:00.421805 5581 catalog_manager.cc:1347] Generated new cluster ID: ebc064a1b9ee4e72a45a624c54f4a1a8
I20260926 14:45:00.421912 5581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:00.427452 5581 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:45:00.428742 5581 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:00.435187 5581 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8920cf0c69004597b71d7602712ce823: Generated new TSK 0
I20260926 14:45:00.436075 5581 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 14:45:00.471501 5597 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 14:45:00.471621 5597 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 14:45:00.472076 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:00.447769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56642 (local address 127.0.98.62:44317)
0926 14:45:00.447913 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:00.447918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:00.448146 (+ 228us) server_negotiation.cc:407] Connection header received
0926 14:45:00.448372 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:00.448374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:00.448424 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:00.448514 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:45:00.448807 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.449326 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.450629 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.450912 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.451000 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:00.451404 (+ 404us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:45:00.471668 (+ 20264us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 14:45:00.471954 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56642: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 14:45:00.472160 5570 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56642: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 14:45:00.473543 5596 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:00.447665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44317 (local address 127.0.0.1:56642)
0926 14:45:00.448037 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:00.448054 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:00.448203 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:00.448544 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:00.448547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:00.448559 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:45:00.448700 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:00.448706 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.449469 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.449473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:00.450485 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:00.450497 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.451007 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.451031 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:00.451126 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:00.451188 (+ 62us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:45:00.471932 (+ 20744us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 14:45:00.472191 (+ 259us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:44317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:44317: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 14:45:00.474242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 5500
2026-09-26T14:45:00Z chronyd exiting
I20260926 14:45:00.510895 392 test_util.cc:175] -----------------------------------------------
I20260926 14:45:00.510985 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790433872429009-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (319 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 14:45:00.512005 392 mini_oidc.cc:152] Starting JWKS server
I20260926 14:45:00.512849 392 webserver.cc:460] Webserver started at https://127.0.0.1:43839/ using document root <none> and password file <none>
I20260926 14:45:00.512930 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:45:00.513139 392 webserver.cc:460] Webserver started at http://127.0.0.1:36103/ using document root <none> and password file <none>
2026-09-26T14:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:00Z Disabled control of system clock
I20260926 14:45:00.521623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40539
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40539
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43839/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/testchainca.cert with env {}
W20260926 14:45:00.630389 5603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:00.630697 5603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:00.630738 5603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:00.634977 5603 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:00.635056 5603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:00.635069 5603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:00.635084 5603 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:00.635099 5603 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:45:00.635114 5603 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:45:00.635131 5603 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43839/jwks.json
I20260926 14:45:00.640911 5603 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43839/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:00.642071 5603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:00.643191 5603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:00.651822 5609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:00.651964 5603 server_base.cc:1034] running on GCE node
W20260926 14:45:00.651786 5608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:00.652139 5611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:00.652714 5603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:00.653898 5603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:00.655057 5603 hybrid_clock.cc:648] HybridClock initialized: now 1790433900655034 us; error 37 us; skew 500 ppm
I20260926 14:45:00.657040 5603 webserver.cc:460] Webserver started at http://127.0.98.62:46807/ using document root <none> and password file <none>
I20260926 14:45:00.657613 5603 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:00.657681 5603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:00.657873 5603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:00.660007 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "7683ec63b531486c858a0853229402d4"
format_stamp: "Formatted at 2026-09-26 14:45:00 on dist-test-slave-s7k6"
server_key: "928fc1c8b07ce4e8a8cc7b75357e7645"
server_key_iv: "28b9d623726869ff98751fecc6eba666"
server_key_version: "encryptionkey@0"
I20260926 14:45:00.660583 5603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "7683ec63b531486c858a0853229402d4"
format_stamp: "Formatted at 2026-09-26 14:45:00 on dist-test-slave-s7k6"
server_key: "928fc1c8b07ce4e8a8cc7b75357e7645"
server_key_iv: "28b9d623726869ff98751fecc6eba666"
server_key_version: "encryptionkey@0"
I20260926 14:45:00.664355 5603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:45:00.666909 5617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:00.668344 5603 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:00.668466 5603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "7683ec63b531486c858a0853229402d4"
format_stamp: "Formatted at 2026-09-26 14:45:00 on dist-test-slave-s7k6"
server_key: "928fc1c8b07ce4e8a8cc7b75357e7645"
server_key_iv: "28b9d623726869ff98751fecc6eba666"
server_key_version: "encryptionkey@0"
I20260926 14:45:00.668572 5603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:00.693518 5603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:00.694245 5603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:00.694418 5603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:00.701043 5603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40539
I20260926 14:45:00.701053 5669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40539 every 8 connection(s)
I20260926 14:45:00.702034 5603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:00.705221 5670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:00.707450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 5603
I20260926 14:45:00.707576 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:45:00.707880 392 external_mini_cluster.cc:1428] Setting key b8a5ebe29a56cec282e6515f1f545c6f
I20260926 14:45:00.711668 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4: Bootstrap starting.
I20260926 14:45:00.714202 5670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:00.715026 5670 log.cc:824] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:00.717160 5670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4: No bootstrap required, opened a new log
I20260926 14:45:00.717458 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:00.709342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56426 (local address 127.0.98.62:40539)
0926 14:45:00.710033 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:00.710049 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:00.710087 (+ 38us) server_negotiation.cc:407] Connection header received
0926 14:45:00.710816 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:00.710843 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:00.711290 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:00.711630 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:00.712084 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.713146 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.714057 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.714576 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.714736 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:00.715619 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:00.715625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:00.715639 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:00.716100 (+ 461us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:00.716159 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:00.716179 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:00.716265 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:00.716404 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 14:45:00.716680 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":126,"threads_started":1}
I20260926 14:45:00.720556 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:45:00.721376 5670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7683ec63b531486c858a0853229402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } }
I20260926 14:45:00.721647 5670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:00.721715 5670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7683ec63b531486c858a0853229402d4, State: Initialized, Role: FOLLOWER
I20260926 14:45:00.722203 5670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7683ec63b531486c858a0853229402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } }
I20260926 14:45:00.722376 5670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:00.722440 5670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:00.722514 5670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:00.723475 5670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7683ec63b531486c858a0853229402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } }
I20260926 14:45:00.723816 5670 leader_election.cc:302] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7683ec63b531486c858a0853229402d4; no voters:
I20260926 14:45:00.724099 5670 leader_election.cc:288] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:45:00.724189 5675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:00.724416 5675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [term 1 LEADER]: Becoming Leader. State: Replica: 7683ec63b531486c858a0853229402d4, State: Running, Role: LEADER
I20260926 14:45:00.724735 5675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7683ec63b531486c858a0853229402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } }
I20260926 14:45:00.725301 5670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:00.726320 5676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7683ec63b531486c858a0853229402d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7683ec63b531486c858a0853229402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } } }
I20260926 14:45:00.726449 5676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:00.726728 5677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7683ec63b531486c858a0853229402d4. Latest consensus state: current_term: 1 leader_uuid: "7683ec63b531486c858a0853229402d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7683ec63b531486c858a0853229402d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40539 } } }
I20260926 14:45:00.726828 5684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:00.726848 5677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:00.729784 5684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:00.735730 5684 catalog_manager.cc:1347] Generated new cluster ID: 7069e783789d4ab89ef58520ef562793
I20260926 14:45:00.735800 5684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:00.748534 5684 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:45:00.749859 5684 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:00.756524 5684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7683ec63b531486c858a0853229402d4: Generated new TSK 0
I20260926 14:45:00.757395 5684 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 14:45:00.787125 5700 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 14:45:00.787206 5700 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 14:45:00.787952 5673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:00.771568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.0.98.62:40539)
0926 14:45:00.771693 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:00.771697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:00.771801 (+ 104us) server_negotiation.cc:407] Connection header received
0926 14:45:00.772012 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:00.772015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:00.772077 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:00.772209 (+ 132us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:45:00.772520 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.773402 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.774266 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.774611 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.774709 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:00.775106 (+ 397us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:45:00.787535 (+ 12429us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 14:45:00.787812 (+ 277us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 14:45:00.788025 5673 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56436: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 14:45:00.788030 5699 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:00.771446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40539 (local address 127.0.0.1:56436)
0926 14:45:00.771698 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:00.771712 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:00.771867 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:00.772289 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:00.772292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:00.772301 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:45:00.772409 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:00.772416 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.773517 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.773520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:00.774155 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:00.774164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:00.774714 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:00.774733 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:00.774901 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:00.774929 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:45:00.787745 (+ 12816us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 14:45:00.787884 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40539: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43839/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260926 14:45:00.788658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 5603
2026-09-26T14:45:00Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (315 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 14:45:00.827258 392 mini_oidc.cc:152] Starting JWKS server
I20260926 14:45:00.828138 392 webserver.cc:460] Webserver started at https://127.0.0.1:41879/ using document root <none> and password file <none>
I20260926 14:45:00.828238 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 14:45:00.828490 392 webserver.cc:460] Webserver started at http://127.0.0.1:41793/ using document root <none> and password file <none>
2026-09-26T14:45:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:00Z Disabled control of system clock
I20260926 14:45:00.837276 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34747
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34747
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41879/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 14:45:00.946975 5706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:00.947274 5706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:00.947314 5706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:00.951363 5706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:00.951440 5706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:00.951457 5706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:00.951480 5706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:00.951496 5706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:45:00.951511 5706 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:45:00.951524 5706 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41879/jwks.json
I20260926 14:45:00.956542 5706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34747
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41879/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:00.957670 5706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:00.959225 5706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:00.967928 5712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:00.967990 5714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:00.968043 5711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:00.968132 5706 server_base.cc:1034] running on GCE node
I20260926 14:45:00.968833 5706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:00.970080 5706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:00.971243 5706 hybrid_clock.cc:648] HybridClock initialized: now 1790433900971217 us; error 45 us; skew 500 ppm
I20260926 14:45:00.977342 5706 webserver.cc:460] Webserver started at http://127.0.98.62:34165/ using document root <none> and password file <none>
I20260926 14:45:00.977978 5706 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:00.978055 5706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:00.978256 5706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:00.980441 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "b0901293c2834fd6ac3d09b90ba3a55c"
format_stamp: "Formatted at 2026-09-26 14:45:00 on dist-test-slave-s7k6"
server_key: "5130a0d79dda521d7b1734c522ec6274"
server_key_iv: "e5f90027c825bcc9c49829ba1ba20920"
server_key_version: "encryptionkey@0"
I20260926 14:45:00.981029 5706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "b0901293c2834fd6ac3d09b90ba3a55c"
format_stamp: "Formatted at 2026-09-26 14:45:00 on dist-test-slave-s7k6"
server_key: "5130a0d79dda521d7b1734c522ec6274"
server_key_iv: "e5f90027c825bcc9c49829ba1ba20920"
server_key_version: "encryptionkey@0"
I20260926 14:45:00.985073 5706 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:45:00.987735 5720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:00.989200 5706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:00.989331 5706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "b0901293c2834fd6ac3d09b90ba3a55c"
format_stamp: "Formatted at 2026-09-26 14:45:00 on dist-test-slave-s7k6"
server_key: "5130a0d79dda521d7b1734c522ec6274"
server_key_iv: "e5f90027c825bcc9c49829ba1ba20920"
server_key_version: "encryptionkey@0"
I20260926 14:45:00.989454 5706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:00.999470 5706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:01.000150 5706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:01.000317 5706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:01.007115 5706 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34747
I20260926 14:45:01.007138 5772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34747 every 8 connection(s)
I20260926 14:45:01.008051 5706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:01.011264 5773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:01.013043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 5706
I20260926 14:45:01.013168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:45:01.013414 392 external_mini_cluster.cc:1428] Setting key 7b1a8afdb7f07837513d1eef08c6485e
I20260926 14:45:01.018904 5773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c: Bootstrap starting.
I20260926 14:45:01.021581 5773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:01.022490 5773 log.cc:824] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:01.024250 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:01.014782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39020 (local address 127.0.98.62:34747)
0926 14:45:01.015615 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:01.015641 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:01.015686 (+ 45us) server_negotiation.cc:407] Connection header received
0926 14:45:01.016553 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:01.016578 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:01.017111 (+ 533us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:01.017489 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:01.017965 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.019438 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.020351 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.021048 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.021216 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:01.022089 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:01.022096 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:01.022112 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:01.022619 (+ 507us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:01.022689 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:01.022715 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:01.022852 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:01.023012 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 14:45:01.023279 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":152,"threads_started":1}
I20260926 14:45:01.024813 5773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c: No bootstrap required, opened a new log
I20260926 14:45:01.027355 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:45:01.028890 5773 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0901293c2834fd6ac3d09b90ba3a55c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34747 } }
I20260926 14:45:01.029148 5773 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:01.029194 5773 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0901293c2834fd6ac3d09b90ba3a55c, State: Initialized, Role: FOLLOWER
I20260926 14:45:01.029637 5773 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0901293c2834fd6ac3d09b90ba3a55c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34747 } }
I20260926 14:45:01.029762 5773 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:01.029822 5773 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:01.029918 5773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:01.030921 5773 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0901293c2834fd6ac3d09b90ba3a55c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34747 } }
I20260926 14:45:01.031258 5773 leader_election.cc:302] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0901293c2834fd6ac3d09b90ba3a55c; no voters:
I20260926 14:45:01.031561 5773 leader_election.cc:288] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:45:01.031693 5778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:01.031916 5778 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [term 1 LEADER]: Becoming Leader. State: Replica: b0901293c2834fd6ac3d09b90ba3a55c, State: Running, Role: LEADER
I20260926 14:45:01.032231 5778 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0901293c2834fd6ac3d09b90ba3a55c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34747 } }
I20260926 14:45:01.032750 5773 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:01.033993 5779 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0901293c2834fd6ac3d09b90ba3a55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0901293c2834fd6ac3d09b90ba3a55c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34747 } } }
I20260926 14:45:01.034134 5779 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:01.034458 5780 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0901293c2834fd6ac3d09b90ba3a55c. Latest consensus state: current_term: 1 leader_uuid: "b0901293c2834fd6ac3d09b90ba3a55c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0901293c2834fd6ac3d09b90ba3a55c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34747 } } }
I20260926 14:45:01.034554 5780 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c [sys.catalog]: This master's current role is: LEADER
W20260926 14:45:01.037211 5793 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:45:01.037281 5793 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:45:01.037353 5789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:01.038205 5789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:01.044083 5789 catalog_manager.cc:1347] Generated new cluster ID: 23e3cb1182a249f3aa7ce5aadb65d95a
I20260926 14:45:01.044167 5789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:01.051717 5789 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:45:01.052920 5789 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:01.064034 5789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0901293c2834fd6ac3d09b90ba3a55c: Generated new TSK 0
I20260926 14:45:01.064785 5789 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:45:01.083448 5776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:01.078044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39030 (local address 127.0.98.62:34747)
0926 14:45:01.078167 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:01.078171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:01.078287 (+ 116us) server_negotiation.cc:407] Connection header received
0926 14:45:01.078445 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:01.078449 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:01.078505 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:01.078625 (+ 120us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:45:01.078925 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.079747 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.080644 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.081018 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.081120 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:01.081463 (+ 343us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:45:01.083046 (+ 1583us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41879/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 14:45:01.083272 (+ 226us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41879/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 14:45:01.083525 5776 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41879/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 14:45:01.085255 5802 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:01.077914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34747 (local address 127.0.0.1:39030)
0926 14:45:01.078205 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:01.078217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:01.078316 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:01.078657 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:01.078660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:01.078669 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:45:01.078817 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:01.078825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.079870 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.079874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:01.080539 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:01.080547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.081149 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.081171 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:01.081248 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:01.081274 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:45:01.083240 (+ 1966us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41879/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 14:45:01.085077 (+ 1837us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34747: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41879/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":1568,"thread_start_us":105,"threads_started":1}
I20260926 14:45:01.086247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 5706
E20260926 14:45:01.093727 5803 webserver.cc:533] Webserver: sslize failed:
E20260926 14:45:01.093802 5803 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T14:45:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (297 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T14:45:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:01Z Disabled control of system clock
I20260926 14:45:01.132623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33405
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33405
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/testchainca.cert with env {}
W20260926 14:45:01.243415 5807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:01.243705 5807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:01.243744 5807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:01.247908 5807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:01.247975 5807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:01.247990 5807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:01.248006 5807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:01.248021 5807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 14:45:01.248037 5807 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 14:45:01.248049 5807 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 14:45:01.253539 5807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33405
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:01.254557 5807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:01.255614 5807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:01.263841 5812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:01.263878 5815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:01.263896 5813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:01.264242 5807 server_base.cc:1034] running on GCE node
I20260926 14:45:01.264782 5807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:01.265892 5807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:01.267062 5807 hybrid_clock.cc:648] HybridClock initialized: now 1790433901267043 us; error 35 us; skew 500 ppm
I20260926 14:45:01.268884 5807 webserver.cc:460] Webserver started at http://127.0.98.62:43517/ using document root <none> and password file <none>
I20260926 14:45:01.269420 5807 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:01.269482 5807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:01.269639 5807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:01.271746 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "3cf42cd3cbf84b7ba05862778f257cb1"
format_stamp: "Formatted at 2026-09-26 14:45:01 on dist-test-slave-s7k6"
server_key: "ec173364322a580ee95fcf26c898ae39"
server_key_iv: "3bd39b09bd77175245045616b6540a9c"
server_key_version: "encryptionkey@0"
I20260926 14:45:01.272264 5807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "3cf42cd3cbf84b7ba05862778f257cb1"
format_stamp: "Formatted at 2026-09-26 14:45:01 on dist-test-slave-s7k6"
server_key: "ec173364322a580ee95fcf26c898ae39"
server_key_iv: "3bd39b09bd77175245045616b6540a9c"
server_key_version: "encryptionkey@0"
I20260926 14:45:01.275924 5807 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 14:45:01.278400 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:01.279660 5807 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:45:01.279767 5807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "3cf42cd3cbf84b7ba05862778f257cb1"
format_stamp: "Formatted at 2026-09-26 14:45:01 on dist-test-slave-s7k6"
server_key: "ec173364322a580ee95fcf26c898ae39"
server_key_iv: "3bd39b09bd77175245045616b6540a9c"
server_key_version: "encryptionkey@0"
I20260926 14:45:01.279863 5807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:01.299877 5807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:01.300577 5807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:01.300737 5807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:01.307366 5807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33405
I20260926 14:45:01.307379 5873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33405 every 8 connection(s)
I20260926 14:45:01.308308 5807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:01.311553 5874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:01.317203 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1: Bootstrap starting.
I20260926 14:45:01.318457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 5807
I20260926 14:45:01.318578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:45:01.318871 392 external_mini_cluster.cc:1428] Setting key c63d194e18007224c375e50ce2b28413
I20260926 14:45:01.320003 5874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:01.320921 5874 log.cc:824] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:01.323239 5874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1: No bootstrap required, opened a new log
I20260926 14:45:01.327633 5874 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf42cd3cbf84b7ba05862778f257cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } }
I20260926 14:45:01.327927 5874 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:01.327996 5874 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cf42cd3cbf84b7ba05862778f257cb1, State: Initialized, Role: FOLLOWER
I20260926 14:45:01.328493 5874 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf42cd3cbf84b7ba05862778f257cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } }
I20260926 14:45:01.328645 5874 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:01.328708 5874 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:01.328796 5874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:01.329114 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:01.319953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.0.98.62:33405)
0926 14:45:01.320680 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:01.320701 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:01.320745 (+ 44us) server_negotiation.cc:407] Connection header received
0926 14:45:01.321603 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:01.321624 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:01.322058 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:01.322428 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:01.322896 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.324332 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.325222 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.325964 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.326139 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:01.326969 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:01.326975 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:01.326993 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:01.327507 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:01.327577 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:01.327603 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:01.327696 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:01.327813 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 14:45:01.328104 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":121,"threads_started":1}
I20260926 14:45:01.329816 5874 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf42cd3cbf84b7ba05862778f257cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } }
I20260926 14:45:01.330154 5874 leader_election.cc:302] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cf42cd3cbf84b7ba05862778f257cb1; no voters:
I20260926 14:45:01.330519 5874 leader_election.cc:288] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:45:01.330612 5879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:01.330847 5879 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [term 1 LEADER]: Becoming Leader. State: Replica: 3cf42cd3cbf84b7ba05862778f257cb1, State: Running, Role: LEADER
I20260926 14:45:01.331192 5879 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf42cd3cbf84b7ba05862778f257cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } }
I20260926 14:45:01.331825 5874 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:01.332307 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:45:01.332849 5881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cf42cd3cbf84b7ba05862778f257cb1. Latest consensus state: current_term: 1 leader_uuid: "3cf42cd3cbf84b7ba05862778f257cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf42cd3cbf84b7ba05862778f257cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } } }
I20260926 14:45:01.332976 5881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:01.333271 5880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cf42cd3cbf84b7ba05862778f257cb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cf42cd3cbf84b7ba05862778f257cb1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33405 } } }
I20260926 14:45:01.333364 5880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1 [sys.catalog]: This master's current role is: LEADER
W20260926 14:45:01.336355 5894 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:45:01.336424 5894 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:45:01.336505 5888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:01.337255 5888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:01.342643 5888 catalog_manager.cc:1347] Generated new cluster ID: 8166b547d0be4500a3a0a7bf9c137e9e
I20260926 14:45:01.342712 5888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:01.353966 5888 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:45:01.354943 5888 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:01.366432 5888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cf42cd3cbf84b7ba05862778f257cb1: Generated new TSK 0
I20260926 14:45:01.367285 5888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:45:01.437613 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:01.381178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.0.98.62:33405)
0926 14:45:01.381313 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:01.381317 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:01.381480 (+ 163us) server_negotiation.cc:407] Connection header received
0926 14:45:01.381652 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:01.381654 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:01.381702 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:01.381782 (+ 80us) server_negotiation.cc:226] Negotiated authn=JWT
0926 14:45:01.382039 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.382573 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.383874 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.384143 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.384230 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:01.384646 (+ 416us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 14:45:01.437113 (+ 52467us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 14:45:01.437457 (+ 344us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53960: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 14:45:01.437698 5903 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:01.381077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33405 (local address 127.0.0.1:53960)
0926 14:45:01.381384 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:01.381398 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:01.381520 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:01.381796 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:01.381799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:01.381806 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0926 14:45:01.381914 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:01.381921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.382672 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.382675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:01.383737 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:01.383748 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:01.384237 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:01.384284 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:01.384395 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:01.384427 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 14:45:01.437354 (+ 52927us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 14:45:01.437550 (+ 196us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:33405: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
W20260926 14:45:01.437706 5877 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53960: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260926 14:45:01.438310 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 5807
2026-09-26T14:45:01Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (331 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:45:01 dist-test-slave-s7k6 krb5kdc[5915](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:45:01 dist-test-slave-s7k6 krb5kdc[5915](info): set up 2 sockets
Sep 26 14:45:01 dist-test-slave-s7k6 krb5kdc[5915](info): commencing operation
krb5kdc: starting...
W20260926 14:45:02.630982 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:45:02 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:45:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:45:02.745699 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35347
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:02.854374 5928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:02.854646 5928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:02.854701 5928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:02.858744 5928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:02.858824 5928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:02.858850 5928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:02.858872 5928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:02.858893 5928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:45:02.864193 5928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:02.865214 5928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:02.866238 5928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:02.874487 5934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:02.874498 5936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:02.874496 5933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:02.875036 5928 server_base.cc:1034] running on GCE node
I20260926 14:45:02.875674 5928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:02.876948 5928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:02.878155 5928 hybrid_clock.cc:648] HybridClock initialized: now 1790433902878129 us; error 47 us; skew 500 ppm
Sep 26 14:45:02 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:02.881330 5928 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:45:02.882290 5928 webserver.cc:460] Webserver started at http://127.0.98.62:36375/ using document root <none> and password file <none>
I20260926 14:45:02.882891 5928 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:02.882967 5928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:02.883179 5928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:02.885365 5928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "482796d4e42141bf9f9273b428d3c3de"
format_stamp: "Formatted at 2026-09-26 14:45:02 on dist-test-slave-s7k6"
server_key: "312515f7ef47898d503231060fb93452"
server_key_iv: "c893571701adb5092923ecb710897507"
server_key_version: "encryptionkey@0"
I20260926 14:45:02.885913 5928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "482796d4e42141bf9f9273b428d3c3de"
format_stamp: "Formatted at 2026-09-26 14:45:02 on dist-test-slave-s7k6"
server_key: "312515f7ef47898d503231060fb93452"
server_key_iv: "c893571701adb5092923ecb710897507"
server_key_version: "encryptionkey@0"
I20260926 14:45:02.889813 5928 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 14:45:02.892438 5943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:02.893823 5928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:45:02.893944 5928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "482796d4e42141bf9f9273b428d3c3de"
format_stamp: "Formatted at 2026-09-26 14:45:02 on dist-test-slave-s7k6"
server_key: "312515f7ef47898d503231060fb93452"
server_key_iv: "c893571701adb5092923ecb710897507"
server_key_version: "encryptionkey@0"
I20260926 14:45:02.894052 5928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:02.907549 5928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:02.910687 5928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:02.910887 5928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:02.917739 5928 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35347
I20260926 14:45:02.918004 5995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35347 every 8 connection(s)
I20260926 14:45:02.918720 5928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:02.921572 5996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:02.921710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 5928
I20260926 14:45:02.921823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:45:02.922084 392 external_mini_cluster.cc:1428] Setting key 1b0f3fddc56da3a77a181b2c25931e78
I20260926 14:45:02.928732 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de: Bootstrap starting.
I20260926 14:45:02.931277 5996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de: Neither blocks nor log segments found. Creating new log.
Sep 26 14:45:02 dist-test-slave-s7k6 krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:45:02.932123 5996 log.cc:824] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:02.934531 5996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de: No bootstrap required, opened a new log
I20260926 14:45:02.937709 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:02.923465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42938 (local address 127.0.98.62:35347)
0926 14:45:02.923877 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:02.923896 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:02.923941 (+ 45us) server_negotiation.cc:407] Connection header received
0926 14:45:02.924907 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:02.924933 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:02.925393 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:02.925777 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:02.926453 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:02.927984 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:02.928841 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:02.929503 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:02.929655 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:02.932309 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:02.932316 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:02.932333 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:02.932363 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:02.934412 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:02.934914 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:02.934921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:02.934928 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:02.935080 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:02.935386 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:02.935389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:02.935391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:02.935894 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:02.936080 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:02.936395 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 14:45:02.936713 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":120,"threads_started":1}
I20260926 14:45:02.939081 5996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482796d4e42141bf9f9273b428d3c3de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } }
I20260926 14:45:02.939307 5996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:02.939343 5996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482796d4e42141bf9f9273b428d3c3de, State: Initialized, Role: FOLLOWER
I20260926 14:45:02.939774 5996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482796d4e42141bf9f9273b428d3c3de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } }
I20260926 14:45:02.939909 5996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:02.939975 5996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:02.940057 5996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:02.940941 5996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482796d4e42141bf9f9273b428d3c3de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } }
I20260926 14:45:02.941264 5996 leader_election.cc:302] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 482796d4e42141bf9f9273b428d3c3de; no voters:
I20260926 14:45:02.941520 5996 leader_election.cc:288] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:45:02.941718 6001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:02.941958 6001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [term 1 LEADER]: Becoming Leader. State: Replica: 482796d4e42141bf9f9273b428d3c3de, State: Running, Role: LEADER
I20260926 14:45:02.942286 6001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482796d4e42141bf9f9273b428d3c3de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } }
I20260926 14:45:02.942689 5996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:02.943997 6002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "482796d4e42141bf9f9273b428d3c3de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482796d4e42141bf9f9273b428d3c3de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } } }
I20260926 14:45:02.944029 6003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 482796d4e42141bf9f9273b428d3c3de. Latest consensus state: current_term: 1 leader_uuid: "482796d4e42141bf9f9273b428d3c3de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482796d4e42141bf9f9273b428d3c3de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35347 } } }
I20260926 14:45:02.944131 6002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:02.944137 6003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:02.944525 6013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:02.947992 6013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:02.953624 6013 catalog_manager.cc:1347] Generated new cluster ID: 40b6ec8e878d4b1aa5d7052c686c5476
I20260926 14:45:02.953693 6013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:02.964761 6013 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:45:02.966073 6013 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:02.974255 6013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 482796d4e42141bf9f9273b428d3c3de: Generated new TSK 0
I20260926 14:45:02.975121 6013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:45:03.009526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35347
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:03.120813 6022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:03.121124 6022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:03.121179 6022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:03.125489 6022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:03.125556 6022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:03.125629 6022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:45:03.131196 6022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6022
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:03.132251 6022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:03.133263 6022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:03.142514 6027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:03.142529 6028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:03.142571 6030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:03.142897 6022 server_base.cc:1034] running on GCE node
I20260926 14:45:03.143370 6022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:03.144053 6022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:03.145252 6022 hybrid_clock.cc:648] HybridClock initialized: now 1790433903145218 us; error 47 us; skew 500 ppm
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:03.148020 6022 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:45:03.148939 6022 webserver.cc:460] Webserver started at http://127.0.98.1:42309/ using document root <none> and password file <none>
I20260926 14:45:03.149466 6022 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:03.149524 6022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:03.149704 6022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:03.151798 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f7881773dd84ad4b1cc21bee7c003c3"
format_stamp: "Formatted at 2026-09-26 14:45:03 on dist-test-slave-s7k6"
server_key: "9aa9255f2cfe64cafbe7ab0f55202b44"
server_key_iv: "8d8bacfe51cd6de81d723a95306d8979"
server_key_version: "encryptionkey@0"
I20260926 14:45:03.152311 6022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f7881773dd84ad4b1cc21bee7c003c3"
format_stamp: "Formatted at 2026-09-26 14:45:03 on dist-test-slave-s7k6"
server_key: "9aa9255f2cfe64cafbe7ab0f55202b44"
server_key_iv: "8d8bacfe51cd6de81d723a95306d8979"
server_key_version: "encryptionkey@0"
I20260926 14:45:03.156201 6022 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:45:03.158697 6037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:03.159988 6022 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:45:03.160087 6022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "8f7881773dd84ad4b1cc21bee7c003c3"
format_stamp: "Formatted at 2026-09-26 14:45:03 on dist-test-slave-s7k6"
server_key: "9aa9255f2cfe64cafbe7ab0f55202b44"
server_key_iv: "8d8bacfe51cd6de81d723a95306d8979"
server_key_version: "encryptionkey@0"
I20260926 14:45:03.160220 6022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:03.171368 6022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:03.174017 6022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:03.174185 6022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:03.174733 6022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:03.175565 6022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:03.175608 6022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:03.175647 6022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:03.175675 6022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:03.183991 6022 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36677
I20260926 14:45:03.184012 6150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36677 every 8 connection(s)
I20260926 14:45:03.184857 6022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:45:03.185443 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 6022
I20260926 14:45:03.185568 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:45:03.185849 392 external_mini_cluster.cc:1428] Setting key b0830f7506d44ee0d1cd81257f0a016e
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:45:03.203678 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:03.186718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41487 (local address 127.0.98.62:35347)
0926 14:45:03.186850 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:03.186854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:03.187992 (+ 1138us) server_negotiation.cc:407] Connection header received
0926 14:45:03.189813 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:03.189818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:03.189873 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:03.189970 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:03.191495 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.192104 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.193780 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.194022 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.194101 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:03.199557 (+ 5456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:03.199560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:03.199562 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:03.199595 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:03.201323 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.201888 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.201892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.201893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.201990 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.202342 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.202346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.202348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.202609 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:03.202711 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:03.203326 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 14:45:03.203484 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:45:03.204421 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:03.187269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35347 (local address 127.0.98.1:41487)
0926 14:45:03.187815 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:03.187873 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:03.189565 (+ 1692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:03.190155 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:03.190165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:03.190654 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:03.191323 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:03.191343 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.192235 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.192239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:03.193642 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:03.193654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.194173 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.194177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:03.194374 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:03.195031 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:03.195054 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:03.199320 (+ 4266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:03.201458 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:03.201463 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:03.201478 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:03.201777 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:03.202078 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:03.202081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:03.202082 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:03.202239 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:03.202719 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:03.202725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:03.203039 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 14:45:03.203212 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 14:45:03.203501 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":133,"threads_started":1}
I20260926 14:45:03.205688 6151 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35347
I20260926 14:45:03.205969 6151 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:03.206651 6151 heartbeater.cc:511] Master 127.0.98.62:35347 requested a full tablet report, sending...
I20260926 14:45:03.208493 5957 ts_manager.cc:194] Registered new tserver with Master: 8f7881773dd84ad4b1cc21bee7c003c3 (127.0.98.1:36677)
I20260926 14:45:03.209707 5957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41487
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:45:03.213822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35347
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:03.321743 6156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:03.322029 6156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:03.322072 6156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:03.326295 6156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:03.326372 6156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:03.326432 6156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:45:03.331868 6156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:03.332893 6156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:03.333868 6156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:03.343566 6162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:03.343684 6164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:03.343567 6161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:03.343669 6156 server_base.cc:1034] running on GCE node
I20260926 14:45:03.344352 6156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:03.344975 6156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:03.346148 6156 hybrid_clock.cc:648] HybridClock initialized: now 1790433903346098 us; error 62 us; skew 500 ppm
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:03.348945 6156 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:45:03.349882 6156 webserver.cc:460] Webserver started at http://127.0.98.2:37899/ using document root <none> and password file <none>
I20260926 14:45:03.350445 6156 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:03.350510 6156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:03.350696 6156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:03.352751 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "51c06e4a8fbc4c4c8a94c69c81b323b0"
format_stamp: "Formatted at 2026-09-26 14:45:03 on dist-test-slave-s7k6"
server_key: "bd85b17cdaa424411c6e98a105f923fc"
server_key_iv: "d6afbe9e5f99315c4e32477131757a41"
server_key_version: "encryptionkey@0"
I20260926 14:45:03.353291 6156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "51c06e4a8fbc4c4c8a94c69c81b323b0"
format_stamp: "Formatted at 2026-09-26 14:45:03 on dist-test-slave-s7k6"
server_key: "bd85b17cdaa424411c6e98a105f923fc"
server_key_iv: "d6afbe9e5f99315c4e32477131757a41"
server_key_version: "encryptionkey@0"
I20260926 14:45:03.357115 6156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:45:03.359751 6171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:03.361047 6156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:45:03.361179 6156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "51c06e4a8fbc4c4c8a94c69c81b323b0"
format_stamp: "Formatted at 2026-09-26 14:45:03 on dist-test-slave-s7k6"
server_key: "bd85b17cdaa424411c6e98a105f923fc"
server_key_iv: "d6afbe9e5f99315c4e32477131757a41"
server_key_version: "encryptionkey@0"
I20260926 14:45:03.361323 6156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:03.373883 6156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:03.376602 6156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:03.376772 6156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:03.377316 6156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:03.378094 6156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:03.378140 6156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:03.378199 6156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:03.378227 6156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:03.386483 6156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34237
I20260926 14:45:03.386507 6284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34237 every 8 connection(s)
I20260926 14:45:03.387409 6156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:45:03.389689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 6156
I20260926 14:45:03.389804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:45:03.390092 392 external_mini_cluster.cc:1428] Setting key 97af9b56f08e0e6b3644b28b2fd309d6
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:45:03.402474 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:03.389133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53399 (local address 127.0.98.62:35347)
0926 14:45:03.389315 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:03.389320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:03.390237 (+ 917us) server_negotiation.cc:407] Connection header received
0926 14:45:03.391982 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:03.391985 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:03.392036 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:03.392136 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:03.393332 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.393853 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.395472 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.395726 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.395813 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:03.398939 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:03.398944 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:03.398948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:03.398978 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:03.400377 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.400908 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.400911 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.400913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.400987 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.401320 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.401323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.401325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.401539 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:03.401629 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:03.402171 (+ 542us) server_negotiation.cc:299] Negotiation successful
0926 14:45:03.402298 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:45:03.403364 6287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:03.389604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35347 (local address 127.0.98.2:53399)
0926 14:45:03.390094 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:03.390137 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:03.391725 (+ 1588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:03.392271 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:03.392277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:03.392606 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:03.393184 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:03.393203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.393973 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.393977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:03.395316 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:03.395328 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.395886 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.395891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:03.396095 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:03.396831 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:03.396854 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:03.398674 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:03.400527 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:03.400532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:03.400546 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:03.400802 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:03.401067 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:03.401069 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:03.401071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:03.401221 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:03.401635 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:03.401642 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:03.401896 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 14:45:03.402099 (+ 203us) client_negotiation.cc:240] Negotiation successful
0926 14:45:03.402461 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":136,"threads_started":1}
I20260926 14:45:03.404577 6285 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35347
I20260926 14:45:03.404841 6285 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:03.405402 6285 heartbeater.cc:511] Master 127.0.98.62:35347 requested a full tablet report, sending...
I20260926 14:45:03.406648 5957 ts_manager.cc:194] Registered new tserver with Master: 51c06e4a8fbc4c4c8a94c69c81b323b0 (127.0.98.2:34237)
I20260926 14:45:03.407491 5957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:45:03.417814 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35347
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:03.527122 6290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:03.527412 6290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:03.527456 6290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:03.531481 6290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:03.531543 6290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:03.531603 6290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:45:03.536909 6290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:03.537900 6290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:03.538908 6290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:03.548369 6298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:03.548374 6295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:03.548373 6296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:03.548516 6290 server_base.cc:1034] running on GCE node
I20260926 14:45:03.549047 6290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:03.549705 6290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:03.550819 6290 hybrid_clock.cc:648] HybridClock initialized: now 1790433903550797 us; error 31 us; skew 500 ppm
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:03.553422 6290 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:45:03.554318 6290 webserver.cc:460] Webserver started at http://127.0.98.3:34947/ using document root <none> and password file <none>
I20260926 14:45:03.554903 6290 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:03.554972 6290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:03.555181 6290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:03.557292 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "5e47d7f79063477eae4d0b8e45caef7a"
format_stamp: "Formatted at 2026-09-26 14:45:03 on dist-test-slave-s7k6"
server_key: "5d8b89c10bef5a4ff2a48a6bf66c65e5"
server_key_iv: "80c0abef2ba200533b5fd82799ee0f75"
server_key_version: "encryptionkey@0"
I20260926 14:45:03.557857 6290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5e47d7f79063477eae4d0b8e45caef7a"
format_stamp: "Formatted at 2026-09-26 14:45:03 on dist-test-slave-s7k6"
server_key: "5d8b89c10bef5a4ff2a48a6bf66c65e5"
server_key_iv: "80c0abef2ba200533b5fd82799ee0f75"
server_key_version: "encryptionkey@0"
I20260926 14:45:03.561689 6290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:45:03.564388 6305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:03.565745 6290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:45:03.565846 6290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "5e47d7f79063477eae4d0b8e45caef7a"
format_stamp: "Formatted at 2026-09-26 14:45:03 on dist-test-slave-s7k6"
server_key: "5d8b89c10bef5a4ff2a48a6bf66c65e5"
server_key_iv: "80c0abef2ba200533b5fd82799ee0f75"
server_key_version: "encryptionkey@0"
I20260926 14:45:03.565958 6290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:03.589597 6290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:03.592504 6290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:03.592674 6290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:03.593262 6290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:03.594094 6290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:03.594138 6290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:03.594177 6290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:03.594192 6290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:03.602943 6290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33615
I20260926 14:45:03.602974 6418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33615 every 8 connection(s)
I20260926 14:45:03.603878 6290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:45:03.614074 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 6290
I20260926 14:45:03.614178 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:45:03.614434 392 external_mini_cluster.cc:1428] Setting key 77a1a3eb21c57065d88ea041dc464fcf
I20260926 14:45:03.617291 5999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:03.605699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40311 (local address 127.0.98.62:35347)
0926 14:45:03.605821 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:03.605825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:03.606666 (+ 841us) server_negotiation.cc:407] Connection header received
0926 14:45:03.608017 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:03.608021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:03.608078 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:03.608147 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:03.609371 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.609899 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.611067 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.611292 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.611379 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:03.613825 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:03.613830 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:03.613833 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:03.613863 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:03.615339 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.615881 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.615884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.615885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.615961 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.616298 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.616300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.616301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.616494 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:03.616584 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:03.617071 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 14:45:03.617177 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:45:03.618062 6421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:03.606041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35347 (local address 127.0.98.3:40311)
0926 14:45:03.606489 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:03.606532 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:03.607803 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:03.608334 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:03.608341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:03.608694 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:03.609227 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:03.609241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.610012 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.610015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:03.610952 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:03.610963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.611423 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.611427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:03.611586 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:03.612113 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:03.612198 (+ 85us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:03.613647 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:03.615442 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:03.615447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:03.615459 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:03.615785 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:03.616038 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:03.616042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:03.616044 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:03.616219 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:03.616598 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:03.616604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:03.616855 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 14:45:03.617009 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 14:45:03.617259 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":112,"threads_started":1}
I20260926 14:45:03.619081 6419 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35347
I20260926 14:45:03.619313 6419 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:03.619771 6419 heartbeater.cc:511] Master 127.0.98.62:35347 requested a full tablet report, sending...
I20260926 14:45:03.620723 5957 ts_manager.cc:194] Registered new tserver with Master: 5e47d7f79063477eae4d0b8e45caef7a (127.0.98.3:33615)
I20260926 14:45:03.621322 5957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40311
I20260926 14:45:03.628159 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/test-admin.keytab.
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:03.643070 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 14:45:03.645912 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:03.644522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.0.98.1:36677)
0926 14:45:03.644855 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:03.644868 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:03.644893 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:45:03.644946 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:03.644951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:03.645140 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:03.645313 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:03.645750 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38872: BlockingRecv error: recv got EOF from 127.0.0.1:38872 (error 108)
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":109,"threads_started":1}
W20260926 14:45:03.646021 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:03.647047 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:03.647945 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 14:45:03.656868 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:03.648752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.0.98.1:36677)
0926 14:45:03.648861 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:03.648865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:03.648945 (+ 80us) server_negotiation.cc:407] Connection header received
0926 14:45:03.649072 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:03.649075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:03.649116 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:03.649174 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:03.649678 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.650446 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.651301 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.651673 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.651764 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:03.653778 (+ 2014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:03.653786 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:03.653798 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:03.653823 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:03.655219 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.655638 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.655644 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.655645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.655720 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.655960 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.655963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.655965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.656199 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:03.656410 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:03.656583 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 14:45:03.656724 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 14:45:03.660516 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:03.659583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38884 (local address 127.0.98.1:36677)
0926 14:45:03.659697 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:03.659701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:03.659770 (+ 69us) server_negotiation.cc:407] Connection header received
0926 14:45:03.659887 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:03.659890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:03.659929 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:03.659987 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:03.660420 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38884: BlockingRecv error: recv got EOF from 127.0.0.1:38884 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 14:45:03.660624 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:03.661556 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:03.662318 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 14:45:03.670008 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:03.663065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.0.98.1:36677)
0926 14:45:03.663174 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:03.663177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:03.663274 (+ 97us) server_negotiation.cc:407] Connection header received
0926 14:45:03.663355 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:03.663357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:03.663387 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:03.663457 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:03.663870 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.664434 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.665165 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.665445 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.665522 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:03.667508 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:03.667511 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:03.667513 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:03.667533 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:03.668643 (+ 1110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.669009 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.669011 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.669012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.669081 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.669301 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.669303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.669304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.669446 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:03.669574 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:03.669760 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 14:45:03.669860 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 14:45:03.672417 6426 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:03.671543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.0.98.1:36677)
0926 14:45:03.671660 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:03.671664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:03.671786 (+ 122us) server_negotiation.cc:407] Connection header received
0926 14:45:03.671906 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:03.671908 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:03.671939 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:03.671995 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:03.672322 (+ 327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38912: BlockingRecv error: recv got EOF from 127.0.0.1:38912 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 14:45:03.672487 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790433872429009-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:03.673420 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:03.674180 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[5915](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 14:45:03.681862 6426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:03.674968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38914 (local address 127.0.98.1:36677)
0926 14:45:03.675081 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:03.675084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:03.675135 (+ 51us) server_negotiation.cc:407] Connection header received
0926 14:45:03.675217 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:03.675220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:03.675285 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:03.675350 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:03.675794 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.676305 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.677091 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:03.677302 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:03.677378 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:03.679430 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:03.679433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:03.679434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:03.679455 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:03.680669 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.681010 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.681012 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.681013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.681077 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:03.681283 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:03.681285 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:03.681286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:03.681427 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:03.681518 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:03.681665 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 14:45:03.681751 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:45:03.682737 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 6022
I20260926 14:45:03.687642 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 6156
I20260926 14:45:03.692852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 6290
I20260926 14:45:03.697220 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 5928
2026-09-26T14:45:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2017 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[6439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[6439](info): set up 2 sockets
Sep 26 14:45:03 dist-test-slave-s7k6 krb5kdc[6439](info): commencing operation
krb5kdc: starting...
W20260926 14:45:04.650965 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:45:04 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:45:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 14:45:04.767681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38541
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:04.876729 6452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:04.877027 6452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:04.877067 6452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:04.881146 6452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:04.881217 6452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:04.881235 6452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:04.881254 6452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:04.881299 6452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:45:04.886790 6452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:04.887892 6452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:04.888934 6452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:04.897042 6458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:04.897053 6460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:04.897149 6452 server_base.cc:1034] running on GCE node
W20260926 14:45:04.897053 6457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:04.897791 6452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:04.898933 6452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:04.900090 6452 hybrid_clock.cc:648] HybridClock initialized: now 1790433904900065 us; error 38 us; skew 500 ppm
Sep 26 14:45:04 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433904, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:04.902805 6452 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 14:45:04.903815 6452 webserver.cc:460] Webserver started at http://127.0.98.62:41411/ using document root <none> and password file <none>
I20260926 14:45:04.904369 6452 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:04.904438 6452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:04.904635 6452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:04.906644 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc"
format_stamp: "Formatted at 2026-09-26 14:45:04 on dist-test-slave-s7k6"
server_key: "edf1db0b2925e08f4b9642873f35d905"
server_key_iv: "49e3a6062462217339cfb344f2766665"
server_key_version: "encryptionkey@0"
I20260926 14:45:04.907258 6452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc"
format_stamp: "Formatted at 2026-09-26 14:45:04 on dist-test-slave-s7k6"
server_key: "edf1db0b2925e08f4b9642873f35d905"
server_key_iv: "49e3a6062462217339cfb344f2766665"
server_key_version: "encryptionkey@0"
I20260926 14:45:04.911031 6452 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:45:04.913609 6467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:04.914916 6452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:45:04.915035 6452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc"
format_stamp: "Formatted at 2026-09-26 14:45:04 on dist-test-slave-s7k6"
server_key: "edf1db0b2925e08f4b9642873f35d905"
server_key_iv: "49e3a6062462217339cfb344f2766665"
server_key_version: "encryptionkey@0"
I20260926 14:45:04.915138 6452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:04.937623 6452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:04.940970 6452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:04.941155 6452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:04.947964 6452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38541
I20260926 14:45:04.948248 6519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38541 every 8 connection(s)
I20260926 14:45:04.948930 6452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:04.951764 6520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:04.953886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 6452
I20260926 14:45:04.953992 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:45:04.954238 392 external_mini_cluster.cc:1428] Setting key c7dbf121030fcaa561bc68ad151ff32f
I20260926 14:45:04.957841 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Bootstrap starting.
I20260926 14:45:04.960469 6520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:04.961292 6520 log.cc:824] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:04.963663 6520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: No bootstrap required, opened a new log
Sep 26 14:45:04 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 14:45:04.966795 6520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } }
I20260926 14:45:04.967093 6520 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:04.967157 6520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60eabd2bd7fd43d7b3dfc3545f3fbbfc, State: Initialized, Role: FOLLOWER
I20260926 14:45:04.967649 6520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } }
I20260926 14:45:04.967779 6520 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:04.967820 6520 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:04.967913 6520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:04.968968 6520 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } }
I20260926 14:45:04.969280 6520 leader_election.cc:302] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60eabd2bd7fd43d7b3dfc3545f3fbbfc; no voters:
I20260926 14:45:04.969556 6520 leader_election.cc:288] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:45:04.969663 6525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:04.969791 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:04.955613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.0.98.62:38541)
0926 14:45:04.956024 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:04.956044 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:04.956080 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:45:04.956746 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:04.956764 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:04.957146 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:04.957454 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:04.958364 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:04.959430 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:04.960589 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:04.961100 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:04.961234 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:04.964523 (+ 3289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:04.964529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:04.964543 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:04.964570 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:04.966596 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:04.967122 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:04.967128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:04.967134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:04.967297 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:04.967597 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:04.967601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:04.967602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:04.968092 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:04.968255 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:04.968531 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 14:45:04.968820 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":115,"threads_started":1}
I20260926 14:45:04.969892 6525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 1 LEADER]: Becoming Leader. State: Replica: 60eabd2bd7fd43d7b3dfc3545f3fbbfc, State: Running, Role: LEADER
I20260926 14:45:04.970353 6525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } }
I20260926 14:45:04.970803 6520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:04.971966 6527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60eabd2bd7fd43d7b3dfc3545f3fbbfc. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } }
I20260926 14:45:04.972374 6527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:04.972271 6526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } }
I20260926 14:45:04.972678 6526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: This master's current role is: LEADER
W20260926 14:45:04.975523 6540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:45:04.975595 6540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:45:04.975688 6539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:04.976593 6539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:04.982180 6539 catalog_manager.cc:1347] Generated new cluster ID: 97faa1410e51424e899ad395ff4fefd7
I20260926 14:45:04.982251 6539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:04.991359 6539 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:45:04.992302 6539 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:05.005856 6539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Generated new TSK 0
I20260926 14:45:05.006577 6539 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 14:45:05.037417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38541
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:05.158830 6546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:05.159134 6546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:05.159194 6546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:05.164299 6546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:05.164441 6546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:05.164543 6546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:45:05.170080 6546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:05.171226 6546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:05.172251 6546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:45:05.181597 6546 server_base.cc:1034] running on GCE node
W20260926 14:45:05.181561 6552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:05.181561 6554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:05.181558 6551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:05.182219 6546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:05.182924 6546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:05.184096 6546 hybrid_clock.cc:648] HybridClock initialized: now 1790433905184061 us; error 44 us; skew 500 ppm
Sep 26 14:45:05 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:05.186805 6546 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 14:45:05.187638 6546 webserver.cc:460] Webserver started at http://127.0.98.1:38427/ using document root <none> and password file <none>
I20260926 14:45:05.188136 6546 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:05.188194 6546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:05.188349 6546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:05.190292 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "082247f502db4b6d88447f749061b795"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "6e6729aa5ecc6c6e249443698f352035"
server_key_iv: "c317f559c452902c78efa5983e428836"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.190842 6546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "082247f502db4b6d88447f749061b795"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "6e6729aa5ecc6c6e249443698f352035"
server_key_iv: "c317f559c452902c78efa5983e428836"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.194343 6546 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:05.196926 6561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:05.198247 6546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:45:05.198356 6546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "082247f502db4b6d88447f749061b795"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "6e6729aa5ecc6c6e249443698f352035"
server_key_iv: "c317f559c452902c78efa5983e428836"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.198508 6546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:05.216336 6546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:05.219707 6546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:05.219880 6546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:05.220466 6546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:05.221285 6546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:05.221349 6546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:05.221390 6546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:05.221426 6546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:05.230063 6546 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35681
I20260926 14:45:05.230096 6674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35681 every 8 connection(s)
I20260926 14:45:05.230955 6546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:45:05.233996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 6546
I20260926 14:45:05.234118 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:45:05.234436 392 external_mini_cluster.cc:1428] Setting key 444d038074e646440ebe6943a51f0a1f
Sep 26 14:45:05 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 14:45:05.246649 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.232783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40157 (local address 127.0.98.62:38541)
0926 14:45:05.232915 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:05.232920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:05.233876 (+ 956us) server_negotiation.cc:407] Connection header received
0926 14:45:05.235550 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:05.235554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:05.235609 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:05.235686 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:05.237344 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.237951 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.239492 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.239816 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.239928 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.242858 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:05.242862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:05.242864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:05.242892 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:05.244528 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.245035 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.245038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.245039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.245125 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.245482 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.245485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.245486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.245683 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:05.245767 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:05.246382 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 14:45:05.246494 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260926 14:45:05.247546 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.233171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.1:40157)
0926 14:45:05.233679 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:05.233740 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:05.235320 (+ 1580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:05.235876 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:05.235887 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:05.236440 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:05.237144 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.237165 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.238099 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.238103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.239345 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.239356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.239983 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.239986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.240145 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.240752 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:05.240773 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:05.242602 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:05.244644 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.244650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.244664 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.244931 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.245214 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.245217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.245218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.245358 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.245770 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:05.245776 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:05.246079 (+ 303us) client_negotiation.cc:769] Sending connection context
0926 14:45:05.246266 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 14:45:05.246581 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":162,"threads_started":1}
I20260926 14:45:05.248880 6675 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38541
I20260926 14:45:05.249166 6675 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:05.249738 6675 heartbeater.cc:511] Master 127.0.98.62:38541 requested a full tablet report, sending...
I20260926 14:45:05.251389 6484 ts_manager.cc:194] Registered new tserver with Master: 082247f502db4b6d88447f749061b795 (127.0.98.1:35681)
I20260926 14:45:05.252534 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40157
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 14:45:05.262353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38541
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:05.371271 6680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:05.371564 6680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:05.371624 6680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:05.375711 6680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:05.375779 6680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:05.375859 6680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:45:05.381503 6680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:05.382560 6680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:05.383608 6680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:05.392880 6686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:05.392894 6685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:05.392894 6688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:05.393076 6680 server_base.cc:1034] running on GCE node
I20260926 14:45:05.393584 6680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:05.394215 6680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:05.395407 6680 hybrid_clock.cc:648] HybridClock initialized: now 1790433905395380 us; error 37 us; skew 500 ppm
Sep 26 14:45:05 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:05.398270 6680 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 14:45:05.399224 6680 webserver.cc:460] Webserver started at http://127.0.98.2:41577/ using document root <none> and password file <none>
I20260926 14:45:05.399765 6680 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:05.399825 6680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:05.400012 6680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:05.402204 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "22f4ab545e034fdbb4cfa9a872941fd9"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "499d9b12fe9072e312adc037572efb9b"
server_key_iv: "4db54656d9b4b48f83dea24a32b43c1c"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.402796 6680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "22f4ab545e034fdbb4cfa9a872941fd9"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "499d9b12fe9072e312adc037572efb9b"
server_key_iv: "4db54656d9b4b48f83dea24a32b43c1c"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.406587 6680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 14:45:05.409288 6695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:05.410859 6680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:05.410991 6680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "22f4ab545e034fdbb4cfa9a872941fd9"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "499d9b12fe9072e312adc037572efb9b"
server_key_iv: "4db54656d9b4b48f83dea24a32b43c1c"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.411098 6680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:05.422502 6680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:05.425776 6680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:05.425951 6680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:05.426503 6680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:05.427335 6680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:05.427380 6680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:05.427429 6680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:05.427457 6680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:05.435883 6680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43973
I20260926 14:45:05.435946 6808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43973 every 8 connection(s)
I20260926 14:45:05.436877 6680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:45:05.438095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 6680
I20260926 14:45:05.438207 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:45:05.438499 392 external_mini_cluster.cc:1428] Setting key 63b7b138d4ba58c93887ea1d7d04d1b1
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 26 14:45:05 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260926 14:45:05.451788 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.438884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42151 (local address 127.0.98.62:38541)
0926 14:45:05.439030 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:05.439034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:05.440073 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 14:45:05.441432 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:05.441435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:05.441492 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:05.441567 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:05.442817 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.443422 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.444636 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.444865 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.444967 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.447704 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:05.447708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:05.447711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:05.447741 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:05.449310 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.449963 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.449967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.449968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.450056 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.450481 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.450485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.450486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.450703 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:05.450884 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:05.451460 (+ 576us) server_negotiation.cc:299] Negotiation successful
0926 14:45:05.451610 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:45:05.452617 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.439416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.2:42151)
0926 14:45:05.439928 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:05.439983 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:05.441204 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:05.441749 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:05.441757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:05.442122 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:05.442629 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.442644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.443554 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.443558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.444509 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.444520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.445016 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.445020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.445234 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.445808 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:05.445826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:05.447515 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:05.449475 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.449482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.449496 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.449839 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.450149 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.450152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.450154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.450356 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.450879 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:05.450887 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:05.451201 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 14:45:05.451387 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 14:45:05.451689 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":118,"threads_started":1}
I20260926 14:45:05.453966 6809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38541
I20260926 14:45:05.454222 6809 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:05.454782 6809 heartbeater.cc:511] Master 127.0.98.62:38541 requested a full tablet report, sending...
I20260926 14:45:05.455888 6484 ts_manager.cc:194] Registered new tserver with Master: 22f4ab545e034fdbb4cfa9a872941fd9 (127.0.98.2:43973)
I20260926 14:45:05.456459 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42151
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 14:45:05.465133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38541
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:05.576380 6814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:05.576659 6814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:05.576702 6814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:05.580852 6814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:05.580915 6814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:05.580981 6814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:45:05.586685 6814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38541
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:05.587766 6814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:05.588783 6814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:05.598142 6820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:05.598187 6814 server_base.cc:1034] running on GCE node
W20260926 14:45:05.598150 6822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:05.598150 6819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:05.598882 6814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:05.599561 6814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:05.600749 6814 hybrid_clock.cc:648] HybridClock initialized: now 1790433905600716 us; error 48 us; skew 500 ppm
Sep 26 14:45:05 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:05.603735 6814 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 14:45:05.604664 6814 webserver.cc:460] Webserver started at http://127.0.98.3:38399/ using document root <none> and password file <none>
I20260926 14:45:05.605247 6814 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:05.605314 6814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:05.605530 6814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:05.607625 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "a195a88278b243b3b6463201766eeb3d"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "a3d270734c4cdb5a87c2d314db3b9c3b"
server_key_iv: "61ee2af32ad8043e403fc5dd71ee43b0"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.608168 6814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a195a88278b243b3b6463201766eeb3d"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "a3d270734c4cdb5a87c2d314db3b9c3b"
server_key_iv: "61ee2af32ad8043e403fc5dd71ee43b0"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.611967 6814 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 14:45:05.614434 6829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:05.615868 6814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:45:05.615988 6814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "a195a88278b243b3b6463201766eeb3d"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "a3d270734c4cdb5a87c2d314db3b9c3b"
server_key_iv: "61ee2af32ad8043e403fc5dd71ee43b0"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.616097 6814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:05.629235 6814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:05.632558 6814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:05.632740 6814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:05.633332 6814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:05.634178 6814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:05.634235 6814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:05.634290 6814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:05.634318 6814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:05.642932 6814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35215
I20260926 14:45:05.642975 6942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35215 every 8 connection(s)
I20260926 14:45:05.643868 6814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:45:05.651338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 6814
I20260926 14:45:05.651454 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:45:05.651738 392 external_mini_cluster.cc:1428] Setting key 89f85a596666f170ade8f93ef111b611
Sep 26 14:45:05 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 14:45:05.657760 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.645738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53549 (local address 127.0.98.62:38541)
0926 14:45:05.645889 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:05.645893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:05.646784 (+ 891us) server_negotiation.cc:407] Connection header received
0926 14:45:05.648120 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:05.648123 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:05.648169 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:05.648250 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:05.649483 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.650067 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.651352 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.651673 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.651785 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.654540 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:05.654543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:05.654545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:05.654570 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:05.655841 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.656316 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.656318 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.656320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.656396 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.656768 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.656771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.656773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.656978 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:05.657063 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:05.657527 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 14:45:05.657638 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:45:05.658581 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.646152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.3:53549)
0926 14:45:05.646600 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:05.646645 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:05.647898 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:05.648401 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:05.648408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:05.648786 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:05.649324 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.649338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.650204 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.650208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.651220 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.651232 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.651823 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.651828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.652012 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.652704 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:05.652727 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:05.654374 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:05.655957 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.655962 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.655975 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.656223 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.656498 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.656501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.656503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.656654 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.657073 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:05.657078 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:05.657315 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 14:45:05.657490 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 14:45:05.657753 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":96,"threads_started":1}
I20260926 14:45:05.659734 6943 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38541
I20260926 14:45:05.659984 6943 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:05.660460 6943 heartbeater.cc:511] Master 127.0.98.62:38541 requested a full tablet report, sending...
I20260926 14:45:05.661554 6484 ts_manager.cc:194] Registered new tserver with Master: a195a88278b243b3b6463201766eeb3d (127.0.98.3:35215)
I20260926 14:45:05.662130 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53549
I20260926 14:45:05.665702 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 14:45:05.691758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43045
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38541,127.0.98.61:43045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:05.804072 6948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:05.804394 6948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:05.804435 6948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:05.808681 6948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:05.808753 6948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:05.808771 6948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:05.808786 6948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:05.808800 6948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:45:05.814360 6948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38541,127.0.98.61:43045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
W20260926 14:45:05.814591 6948 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:38541,127.0.98.61:43045'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 14:45:05.815577 6948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:05.816636 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:05.825109 6956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:05.825107 6954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:05.825253 6948 server_base.cc:1034] running on GCE node
W20260926 14:45:05.825423 6953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:05.825984 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:05.827126 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:05.828301 6948 hybrid_clock.cc:648] HybridClock initialized: now 1790433905828281 us; error 36 us; skew 500 ppm
Sep 26 14:45:05 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:05.831248 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 14:45:05.832221 6948 webserver.cc:460] Webserver started at http://127.0.98.61:36585/ using document root <none> and password file <none>
I20260926 14:45:05.832782 6948 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:05.832856 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:05.833068 6948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:05.835251 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data/instance:
uuid: "8a07669340f7465cba698cb64034b4a1"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "09f13c62ff9b97145cc576bdc3f8f710"
server_key_iv: "c262bead9e4d86a720f30c67844e5e87"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.835841 6948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/wal/instance:
uuid: "8a07669340f7465cba698cb64034b4a1"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "09f13c62ff9b97145cc576bdc3f8f710"
server_key_iv: "c262bead9e4d86a720f30c67844e5e87"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.839908 6948 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:45:05.842660 6963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:05.844206 6948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:45:05.844339 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/wal
uuid: "8a07669340f7465cba698cb64034b4a1"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "09f13c62ff9b97145cc576bdc3f8f710"
server_key_iv: "c262bead9e4d86a720f30c67844e5e87"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.844472 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:05.862294 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:05.865290 6948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:05.865473 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:05.872263 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43045
I20260926 14:45:05.872511 7015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43045 every 8 connection(s)
I20260926 14:45:05.873256 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
I20260926 14:45:05.876442 7016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:05.877744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 6948
I20260926 14:45:05.880913 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:45:05 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 14:45:05 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 14:45:05.891938 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.882657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.0.98.62:38541)
0926 14:45:05.882822 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:05.882827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:05.883141 (+ 314us) server_negotiation.cc:407] Connection header received
0926 14:45:05.883267 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:05.883270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:05.883320 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:05.883407 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:05.884256 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.885065 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.886231 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.886501 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.886584 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.888987 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:05.888990 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:05.888992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:05.889017 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:05.890237 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.890671 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.890674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.890675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.890749 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.891022 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.891025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.891027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.891238 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:05.891327 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:05.891666 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 14:45:05.891779 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:45:05.892706 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.879443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.0.98.61:43045)
0926 14:45:05.879897 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:05.879913 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:05.879956 (+ 43us) server_negotiation.cc:407] Connection header received
0926 14:45:05.880622 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:05.880641 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:05.881002 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:05.881332 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:05.882151 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.883355 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.884525 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.885264 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.885377 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.888156 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:05.888161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:05.888171 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:05.888193 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:05.889850 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.890283 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.890289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.890296 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.890406 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.890655 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.890658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.890660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.891105 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:05.891281 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:05.891528 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 14:45:05.891835 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":126,"threads_started":1}
I20260926 14:45:05.892709 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.882561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:38754)
0926 14:45:05.883009 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:05.883059 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:05.883181 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:05.883470 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:05.883476 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:05.883949 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:05.884109 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.884116 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.885203 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.885206 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.886101 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.886113 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.886656 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.886660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.886785 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.887300 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:05.887316 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:05.888851 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:05.890336 (+ 1485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.890344 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.890350 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.890578 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.890863 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.890866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.890868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.890944 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.891338 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:05.891345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:05.891477 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 14:45:05.891598 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 14:45:05.891927 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":88,"thread_start_us":104,"threads_started":1}
I20260926 14:45:05.894711 7016 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:45:05 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433905, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 14:45:05.902115 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.895260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37980 (local address 127.0.98.61:43045)
0926 14:45:05.895351 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:05.895355 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:05.895367 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:45:05.895462 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:05.895465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:05.895497 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:05.895597 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:05.895932 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.896744 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.897441 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.897657 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.897737 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.899539 (+ 1802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:05.899543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:05.899544 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:05.899563 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:05.900699 (+ 1136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.901049 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.901052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.901054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.901119 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.901330 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.901333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.901335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.901529 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:05.901609 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:05.901832 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 14:45:05.901967 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":35}
I20260926 14:45:05.902115 7020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.895201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43045 (local address 127.0.0.1:37980)
0926 14:45:05.895300 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:05.895313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:05.895390 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:05.895620 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:05.895622 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:05.895763 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:05.895841 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.895846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.896839 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.896842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.897345 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.897352 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.897743 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.897746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.897830 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.898245 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:05.898256 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:05.899422 (+ 1166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:05.900800 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.900803 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.900805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.900971 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.901191 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.901194 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.901196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.901261 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.901628 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:05.901630 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:05.901702 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 14:45:05.901782 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 14:45:05.901930 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 14:45:05.905153 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: Bootstrap starting.
I20260926 14:45:05.907662 7016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:05.908396 7016 log.cc:824] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:05.910266 7016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: No bootstrap required, opened a new log
I20260926 14:45:05.913308 7016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } }
I20260926 14:45:05.913548 7016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:05.913597 7016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a07669340f7465cba698cb64034b4a1, State: Initialized, Role: FOLLOWER
I20260926 14:45:05.914045 7016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } }
I20260926 14:45:05.914542 7022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } } }
I20260926 14:45:05.914659 7022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:45:05.915304 7016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 14:45:05.920161 7033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:45:05.920274 7033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:45:05.921996 6948 master_runner.cc:428] Detected that this master 8a07669340f7465cba698cb64034b4a1 is joining an existing cluster
I20260926 14:45:05.922051 6948 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:43045
I20260926 14:45:05.931874 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.923989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:38760)
0926 14:45:05.924377 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:05.924390 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:05.924505 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:05.924783 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:05.924786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:05.925068 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:05.925212 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.925220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.926240 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.926243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.927145 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.927151 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.927702 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.927706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.927801 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.928213 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:05.928228 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:05.928747 (+ 519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:05.930378 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.930382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.930384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.930683 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.930948 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.930951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.930953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.931036 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.931363 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:05.931366 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:05.931453 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 14:45:05.931554 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 14:45:05.931684 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260926 14:45:05.931907 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.924065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.0.98.62:38541)
0926 14:45:05.924404 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:05.924410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:05.924471 (+ 61us) server_negotiation.cc:407] Connection header received
0926 14:45:05.924620 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:05.924624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:05.924677 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:05.924755 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:05.925323 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.926126 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.927250 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.927588 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.927698 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.928885 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:05.928889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:05.928892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:05.928916 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:05.930281 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.930786 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.930790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.930792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.930873 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.931112 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.931115 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.931117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.931283 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:05.931363 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:05.931638 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 14:45:05.931754 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260926 14:45:05.932200 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.924283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37990 (local address 127.0.98.61:43045)
0926 14:45:05.924476 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:05.924480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:05.924637 (+ 157us) server_negotiation.cc:407] Connection header received
0926 14:45:05.924748 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:05.924751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:05.924786 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:05.924844 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:05.925321 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.925807 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.926727 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.927070 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.927136 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.929000 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:05.929003 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:05.929006 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:05.929028 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:05.930711 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.931168 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.931171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.931173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.931239 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.931458 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.931461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.931463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.931646 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:05.931724 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:05.931970 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 14:45:05.932078 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":142}
I20260926 14:45:05.932487 7039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.924095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43045 (local address 127.0.0.1:37990)
0926 14:45:05.924578 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:05.924589 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:05.924674 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:05.924936 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:05.924938 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:05.925126 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:05.925218 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.925224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.925888 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.925890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.926622 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.926631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.927303 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.927306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.927384 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.928457 (+ 1073us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:05.928471 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:05.928878 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:05.930852 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.930855 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.930857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.931076 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.931321 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.931324 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.931326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.931393 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.931756 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:05.931758 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:05.931827 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 14:45:05.932233 (+ 406us) client_negotiation.cc:240] Negotiation successful
0926 14:45:05.932338 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":26,"thread_start_us":109,"threads_started":1}
Sep 26 14:45:05 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433904, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 14:45:05.945012 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.936548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43045 (local address 127.0.0.1:37992)
0926 14:45:05.936880 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:05.936929 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:05.937061 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:05.937386 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:05.937390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:05.937729 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:05.937851 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.937857 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.938588 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.938591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.939281 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.939290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.939689 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.939692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.939797 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.940303 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:05.940314 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:05.941889 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:05.943523 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.943532 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.943539 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.943759 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.944001 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:05.944004 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:05.944006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:05.944085 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:05.944423 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:05.944430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:05.944550 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 14:45:05.944673 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 14:45:05.944861 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260926 14:45:05.945007 7019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.936613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37992 (local address 127.0.98.61:43045)
0926 14:45:05.936711 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:05.936715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:05.937001 (+ 286us) server_negotiation.cc:407] Connection header received
0926 14:45:05.937197 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:05.937200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:05.937248 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:05.937321 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:05.937980 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.938485 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.939387 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.939597 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.939677 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:05.942047 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:05.942050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:05.942052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:05.942073 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:05.943403 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.943848 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.943852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.943855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.943923 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:05.944158 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:05.944161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:05.944163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:05.944328 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:05.944413 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:05.944733 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 14:45:05.944840 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:45:05.946722 6484 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:43045: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8a07669340f7465cba698cb64034b4a1"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 43045
}
attrs {
promote: true
}
}
dest_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc"
cas_config_opid_index: -1
I20260926 14:45:05.947469 6484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } }
I20260926 14:45:05.950034 6990 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:05.951463 6990 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 FOLLOWER]: Refusing update from remote peer 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 14:45:05.952203 7043 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260926 14:45:05.955708 7044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8a07669340f7465cba698cb64034b4a1 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260926 14:45:05.958290 7022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60eabd2bd7fd43d7b3dfc3545f3fbbfc. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } } } pending_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } }
I20260926 14:45:05.958446 7022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:45:05.958544 7022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } } } pending_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } }
I20260926 14:45:05.958628 7022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:45:05.960381 7044 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:43045
I20260926 14:45:05.960907 6948 master.cc:537] Master@127.0.98.61:43045 shutting down...
I20260926 14:45:05.960945 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } }
I20260926 14:45:05.961076 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:05.963970 7045 mvcc.cc:205] Tried to move back new op lower bound from 7333617274899619840 to 7333617274759372800. Current Snapshot: MvccSnapshot[applied={T|T < 7333617274797142016}]
I20260926 14:45:05.966837 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } }
I20260926 14:45:05.966960 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:05.967234 6948 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 LEARNER]: Raft consensus shutting down.
I20260926 14:45:05.967514 6948 pending_rounds.cc:62] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: Trying to abort 1 pending ops.
I20260926 14:45:05.967585 6948 pending_rounds.cc:68] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7333617278759976960 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "00000000000000000000000000000000" old_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } } new_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } } }
I20260926 14:45:05.967721 6948 raft_consensus.cc:2894] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 LEARNER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20260926 14:45:05.967870 6948 raft_consensus.cc:2866] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260926 14:45:05.967921 6948 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:45:05.967871 6469 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 14:45:05.967983 6948 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: stopping tablet replica
W20260926 14:45:05.968955 6469 consensus_peers.cc:487] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc -> Peer 8a07669340f7465cba698cb64034b4a1 (127.0.98.61:43045): Couldn't send request to peer 8a07669340f7465cba698cb64034b4a1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 14:45:05.979153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46531
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38541,127.0.98.61:43045,127.0.98.60:46531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:45:05.983805 6948 master.cc:552] Master@127.0.98.61:43045 shutdown complete.
I20260926 14:45:05.983858 6948 master_runner.cc:305] Clearing existing system tablet
I20260926 14:45:05.984081 6948 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:05.984128 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:05.985177 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 14:45:05.985935 7057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:05.986173 6948 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:05.986294 6948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:05.986330 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/wal
uuid: "8a07669340f7465cba698cb64034b4a1"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "09f13c62ff9b97145cc576bdc3f8f710"
server_key_iv: "c262bead9e4d86a720f30c67844e5e87"
server_key_version: "encryptionkey@0"
I20260926 14:45:05.987938 6948 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:05.989841 6948 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 14:45:05.989931 6948 log.cc:1196] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 14:45:05.990216 6948 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: Deleting consensus metadata
I20260926 14:45:05.990370 6948 master_runner.cc:315] Copying system tablet from 127.0.98.62:38541
I20260926 14:45:05.991446 6948 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38541
I20260926 14:45:06.004562 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.992978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38766 (local address 127.0.98.62:38541)
0926 14:45:05.993116 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:05.993121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:05.993317 (+ 196us) server_negotiation.cc:407] Connection header received
0926 14:45:05.993433 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:05.993436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:05.993481 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:05.993549 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:05.994002 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.994541 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.995465 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.995697 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.995789 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.001432 (+ 5643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.001436 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.001438 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.001463 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.002963 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.003396 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.003400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.003402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.003480 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.003731 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.003734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.003736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.003922 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.004007 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.004281 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.004396 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 14:45:06.004578 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:05.992941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:38766)
0926 14:45:05.993226 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:05.993237 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:05.993349 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:05.993575 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:05.993578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:05.993782 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:05.993881 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.993887 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.994644 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.994648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.995339 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:05.995349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:05.995803 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:05.995807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:05.995901 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.000887 (+ 4986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.000901 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.001245 (+ 344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.003092 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.003096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.003098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.003301 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.003563 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.003566 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.003568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.003650 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.004007 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.004010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.004123 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.004213 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.004378 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":85,"threads_started":1}
I20260926 14:45:06.005450 6504 tablet_copy_service.cc:140] P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8a07669340f7465cba698cb64034b4a1 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:38766)
I20260926 14:45:06.005616 6504 tablet_copy_service.cc:161] P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8a07669340f7465cba698cb64034b4a1 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:38766: session id = 8a07669340f7465cba698cb64034b4a1-00000000000000000000000000000000
I20260926 14:45:06.007009 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 14:45:06.008407 6948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 14:45:06.010233 6948 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: tablet copy: Starting download of 0 data blocks...
I20260926 14:45:06.010406 6948 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: tablet copy: Starting download of 1 WAL segments...
I20260926 14:45:06.011999 6948 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 14:45:06.013583 6504 tablet_copy_service.cc:342] P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Request end of tablet copy session 8a07669340f7465cba698cb64034b4a1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:38766
I20260926 14:45:06.013691 6504 tablet_copy_service.cc:433] P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: ending tablet copy session 8a07669340f7465cba698cb64034b4a1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8a07669340f7465cba698cb64034b4a1
I20260926 14:45:06.020820 6948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:06.022586 7067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:06.022614 7064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:06.022908 6948 server_base.cc:1034] running on GCE node
W20260926 14:45:06.022922 7065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:06.023116 6948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:06.023396 6948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:06.024574 6948 hybrid_clock.cc:648] HybridClock initialized: now 1790433906024561 us; error 28 us; skew 500 ppm
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:06.026155 6948 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 14:45:06.026588 6948 webserver.cc:460] Webserver started at http://127.0.98.61:43963/ using document root <none> and password file <none>
I20260926 14:45:06.026804 6948 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:06.026845 6948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:06.027889 6948 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:06.028731 7074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.028941 6948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:06.029011 6948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/wal
uuid: "8a07669340f7465cba698cb64034b4a1"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "09f13c62ff9b97145cc576bdc3f8f710"
server_key_iv: "c262bead9e4d86a720f30c67844e5e87"
server_key_version: "encryptionkey@0"
I20260926 14:45:06.029095 6948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:06.043809 6948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:06.045692 6948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:06.050602 6948 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43045
I20260926 14:45:06.050621 7126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43045 every 8 connection(s)
I20260926 14:45:06.051177 6948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
I20260926 14:45:06.052593 7127 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:45:06.053400 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: Bootstrap starting.
I20260926 14:45:06.059432 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:45:06.059720 7127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: Bootstrap complete.
I20260926 14:45:06.060196 7127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } }
I20260926 14:45:06.060303 7127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8a07669340f7465cba698cb64034b4a1, State: Initialized, Role: LEARNER
I20260926 14:45:06.060400 7127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } }
I20260926 14:45:06.060837 7127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:06.064850 7141 catalog_manager.cc:1259] Loaded cluster ID: 97faa1410e51424e899ad395ff4fefd7
I20260926 14:45:06.064906 7141 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: loading cluster ID for follower catalog manager: success
I20260926 14:45:06.066412 7141 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: acquiring CA information for follower catalog manager: success
I20260926 14:45:06.067546 7141 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:45:06.070498 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } }
I20260926 14:45:06.070668 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: This master's current role is: LEARNER
W20260926 14:45:06.130724 7055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:06.131057 7055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:06.131102 7055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:06.135169 7055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:06.135233 7055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:06.135251 7055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:06.135269 7055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:06.135289 7055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:45:06.140846 7055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38541,127.0.98.61:43045,127.0.98.60:46531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:46531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:06.141979 7055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:06.143071 7055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:06.151523 7146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:06.151620 7149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:06.151566 7147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:06.151710 7055 server_base.cc:1034] running on GCE node
I20260926 14:45:06.152362 7055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:06.153522 7055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:06.154692 7055 hybrid_clock.cc:648] HybridClock initialized: now 1790433906154661 us; error 47 us; skew 500 ppm
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:06.158066 7055 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 14:45:06.159336 7055 webserver.cc:460] Webserver started at http://127.0.98.60:38317/ using document root <none> and password file <none>
I20260926 14:45:06.159958 7055 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:06.160035 7055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:06.160243 7055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:06.162619 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data/instance:
uuid: "42fa761df91a4ca98f4f47806f82fc65"
format_stamp: "Formatted at 2026-09-26 14:45:06 on dist-test-slave-s7k6"
server_key: "143bfbb58e66c8c0ddcf5981dbc2c321"
server_key_iv: "b48171178690b78883f0d09aa7303b85"
server_key_version: "encryptionkey@0"
I20260926 14:45:06.163280 7055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/wal/instance:
uuid: "42fa761df91a4ca98f4f47806f82fc65"
format_stamp: "Formatted at 2026-09-26 14:45:06 on dist-test-slave-s7k6"
server_key: "143bfbb58e66c8c0ddcf5981dbc2c321"
server_key_iv: "b48171178690b78883f0d09aa7303b85"
server_key_version: "encryptionkey@0"
I20260926 14:45:06.167374 7055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:45:06.170123 7156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.171607 7055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:06.171803 7055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/wal
uuid: "42fa761df91a4ca98f4f47806f82fc65"
format_stamp: "Formatted at 2026-09-26 14:45:06 on dist-test-slave-s7k6"
server_key: "143bfbb58e66c8c0ddcf5981dbc2c321"
server_key_iv: "b48171178690b78883f0d09aa7303b85"
server_key_version: "encryptionkey@0"
I20260926 14:45:06.171931 7055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:06.189920 7055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:06.193228 7055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:06.193403 7055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:06.200361 7055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46531
I20260926 14:45:06.200625 7208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46531 every 8 connection(s)
I20260926 14:45:06.201444 7055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
I20260926 14:45:06.204871 7209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:06.206437 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 7055
I20260926 14:45:06.209095 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 14:45:06.220901 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.211200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38780 (local address 127.0.98.62:38541)
0926 14:45:06.211337 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.211342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.211627 (+ 285us) server_negotiation.cc:407] Connection header received
0926 14:45:06.211736 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.211739 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.211787 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.211870 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.212537 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.213055 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.213981 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.214247 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.214352 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.217255 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.217259 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.217261 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.217289 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.218676 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.219500 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.219504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.219506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.219591 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.219929 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.219932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.219934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.220126 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.220201 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.220633 (+ 432us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.220744 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:45:06.221873 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.207889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.0.98.60:46531)
0926 14:45:06.208310 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.208329 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.208373 (+ 44us) server_negotiation.cc:407] Connection header received
0926 14:45:06.209233 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.209258 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.209711 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.210097 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.210865 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.212252 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.213129 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.213712 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.213855 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.216699 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.216706 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.216720 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.216747 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.218679 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.219125 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.219129 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.219135 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.219238 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.219696 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.219699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.219700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.220213 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.220389 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.220626 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.220822 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":275,"thread_start_us":114,"threads_started":1}
I20260926 14:45:06.221873 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.211104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:38780)
0926 14:45:06.211486 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.211535 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.211659 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.211918 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.211922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.212263 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.212385 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.212390 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.213180 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.213182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.213866 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.213875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.214447 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.214451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.214569 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.215079 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.215095 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.217116 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.219143 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.219151 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.219156 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.219394 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.219672 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.219675 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.219677 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.219823 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.220216 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.220223 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.220353 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.220524 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.220882 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":85,"thread_start_us":145,"threads_started":1}
I20260926 14:45:06.223107 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 14:45:06.231117 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.223562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43045 (local address 127.0.0.1:38004)
0926 14:45:06.223709 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.223722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.223784 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.224087 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.224089 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.224250 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.224347 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.224352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.225219 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.225222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.225761 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.225768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.226156 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.226158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.226225 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.226636 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.226650 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.227828 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.229529 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.229531 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.229533 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.229717 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.230025 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.230029 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.230030 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.230118 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.230528 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.230531 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.230696 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.230828 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.230971 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260926 14:45:06.231143 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.223679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38004 (local address 127.0.98.61:43045)
0926 14:45:06.223896 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.223899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.223912 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:45:06.223949 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.223951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.223988 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.224071 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.224447 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.225135 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.225847 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.226080 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.226152 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.227980 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.227983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.227985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.228007 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.229428 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.229814 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.229818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.229819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.229915 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.230231 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.230235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.230237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.230443 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.230521 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.230892 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.231000 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20260926 14:45:06.232872 7209 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 14:45:06.240509 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.233310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46531 (local address 127.0.0.1:51094)
0926 14:45:06.233410 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.233423 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.233487 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.233671 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.233673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.233793 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.233860 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.233864 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.234974 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.234976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.235470 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.235476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.235978 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.235981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.236040 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.236490 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.236500 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.237653 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.239189 (+ 1536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.239191 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.239193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.239371 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.239690 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.239692 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.239694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.239768 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.240111 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.240114 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.240192 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.240279 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.240379 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 14:45:06.240638 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.233369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.0.98.60:46531)
0926 14:45:06.233474 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.233478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.233491 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:45:06.233544 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.233546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.233591 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.233662 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.233945 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.234891 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.235606 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.235882 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.235975 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.237779 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.237783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.237785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.237805 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.239100 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.239516 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.239519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.239521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.239593 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.239848 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.239850 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.239852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.240031 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.240101 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.240367 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.240476 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:45:06.243458 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: Bootstrap starting.
I20260926 14:45:06.245817 7209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:06.246536 7209 log.cc:824] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:06.248518 7209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: No bootstrap required, opened a new log
I20260926 14:45:06.251636 7209 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } }
I20260926 14:45:06.251837 7209 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:06.251870 7209 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42fa761df91a4ca98f4f47806f82fc65, State: Initialized, Role: FOLLOWER
I20260926 14:45:06.252307 7209 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } }
I20260926 14:45:06.252770 7215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } } }
I20260926 14:45:06.252889 7215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:45:06.253456 7209 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:06.255234 6675 heartbeater.cc:503] Master 127.0.98.62:38541 was elected leader, sending a full tablet report...
W20260926 14:45:06.258157 7226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:45:06.258236 7226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:45:06.261598 7055 master_runner.cc:428] Detected that this master 42fa761df91a4ca98f4f47806f82fc65 is joining an existing cluster
I20260926 14:45:06.261651 7055 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:46531
I20260926 14:45:06.277076 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 6546
I20260926 14:45:06.281813 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.270984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38784 (local address 127.0.98.62:38541)
0926 14:45:06.271115 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.271119 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.271388 (+ 269us) server_negotiation.cc:407] Connection header received
0926 14:45:06.271598 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.271602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.271655 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.271708 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.272879 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.273738 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.274944 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.275244 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.275324 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.278272 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.278276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.278279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.278311 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.280028 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.280537 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.280541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.280543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.280652 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.280920 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.280923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.280925 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.281142 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.281233 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.281513 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.281650 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:45:06.281826 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.264200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:38784)
0926 14:45:06.271296 (+ 7096us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.271311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.271468 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.271873 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.271876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.272553 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.272722 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.272731 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.273887 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.273890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.274802 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.274814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.275854 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.275858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.275958 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.277025 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.277039 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.278102 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.280146 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.280150 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.280153 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.280433 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.280742 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.280745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.280747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.280840 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.281227 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.281229 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.281336 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.281450 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.281618 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7009,"thread_start_us":141,"threads_started":1}
I20260926 14:45:06.284359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35681
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38541
--webserver_port=38427
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38541,127.0.98.61:43045
--tserver_master_addrs=127.0.98.62:38541,127.0.98.61:43045,127.0.98.60:46531 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433904, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 14:45:06.311652 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.264348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.0.98.61:43045)
0926 14:45:06.264494 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.264499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.275332 (+ 10833us) server_negotiation.cc:407] Connection header received
0926 14:45:06.275464 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.275467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.275530 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.275605 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.276455 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.277236 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.290868 (+ 13632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.291177 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.291250 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.308221 (+ 16971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.308225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.308228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.308261 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.309769 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.310248 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.310251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.310253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.310330 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.310608 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.310611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.310613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.310849 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.310937 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.311348 (+ 411us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.311478 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:45:06.312124 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.264504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43045 (local address 127.0.0.1:38006)
0926 14:45:06.275228 (+ 10724us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.275244 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.275355 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.275881 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.275883 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.276208 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.276334 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.276341 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.277753 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.277756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.278539 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.278549 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.291381 (+ 12832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.291383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.291472 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.307543 (+ 16071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.307559 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.308035 (+ 476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.309888 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.309891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.309894 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.310133 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.310425 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.310428 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.310430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.310507 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.310977 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.310980 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.311088 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.311174 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.311933 (+ 759us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10642,"thread_start_us":98,"threads_started":1}
I20260926 14:45:06.314112 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.264896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46531 (local address 127.0.0.1:51106)
0926 14:45:06.275801 (+ 10905us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.275815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.275925 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.276214 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.276216 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.307011 (+ 30795us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.307174 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.307186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.308163 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.308166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.309005 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.309018 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.309659 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.309662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.309753 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.310143 (+ 390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.310156 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.310510 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.312685 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.312688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.312690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.312897 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.313140 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.313143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.313145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.313214 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.313613 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.313616 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.313704 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.313788 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.313916 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10835,"thread_start_us":593,"threads_started":1}
I20260926 14:45:06.314205 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.286671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46531 (local address 127.0.0.1:51110)
0926 14:45:06.286873 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.286886 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.286965 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.287423 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.287426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.287616 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.287728 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.287735 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.288847 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.288851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.289699 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.289710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.290388 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.290392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.290491 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.291031 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.291046 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.292658 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.309579 (+ 16921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.309583 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.309586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.309850 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.310428 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.310431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.310434 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.310531 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.313651 (+ 3120us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.313654 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.313766 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.313850 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.314008 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127}
I20260926 14:45:06.314607 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.264899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51106 (local address 127.0.98.60:46531)
0926 14:45:06.265075 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.265079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.275876 (+ 10797us) server_negotiation.cc:407] Connection header received
0926 14:45:06.276007 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.276010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.276045 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.276102 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.307323 (+ 31221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.308042 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.309127 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.309429 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.309526 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.310632 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.310636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.310638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.310671 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.312579 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.312983 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.312986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.312988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.313062 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.313286 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.313289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.313291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.313501 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.313582 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.314348 (+ 766us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.314462 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":76,"spinlock_wait_cycles":768}
I20260926 14:45:06.315572 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.286723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51110 (local address 127.0.98.60:46531)
0926 14:45:06.287164 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.287171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.287183 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:45:06.287246 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.287249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.287298 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.287830 (+ 532us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.287932 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.288713 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.289852 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.290252 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.290713 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.292815 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.292819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.292822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.292847 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.309433 (+ 16586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.310227 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.310230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.310232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.310323 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.310640 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.310643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.310645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.313522 (+ 2877us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.315147 (+ 1625us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.315222 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.315335 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":378,"spinlock_wait_cycles":3712,"thread_start_us":117,"threads_started":1}
I20260926 14:45:06.316789 6483 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:46531: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 46531
}
attrs {
promote: true
}
}
dest_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc"
cas_config_opid_index: 5
I20260926 14:45:06.317400 6483 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } }
I20260926 14:45:06.318477 7043 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 42fa761df91a4ca98f4f47806f82fc65 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.319942 7183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:06.320118 7043 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:46531
I20260926 14:45:06.320312 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.320406 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:06.321355 7183 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 FOLLOWER]: Refusing update from remote peer 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 14:45:06.322165 7055 master.cc:537] Master@127.0.98.60:46531 shutting down...
I20260926 14:45:06.322158 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:45:06.323983 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc -> Peer 42fa761df91a4ca98f4f47806f82fc65 (127.0.98.60:46531): Couldn't send request to peer 42fa761df91a4ca98f4f47806f82fc65. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 14:45:06.328081 7055 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:06.328393 7055 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:06.328490 7055 tablet_replica.cc:330] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: stopping tablet replica
I20260926 14:45:06.328648 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.318715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43045 (local address 127.0.0.1:38008)
0926 14:45:06.319245 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.319258 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.319332 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.324561 (+ 5229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.324564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.324586 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:06.324678 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.324684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.325721 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.325725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.327026 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.327039 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.328183 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.328187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.328290 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.328348 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.328420 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.328505 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452}
I20260926 14:45:06.329491 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.318973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38008 (local address 127.0.98.61:43045)
0926 14:45:06.324185 (+ 5212us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.324190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.324206 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:45:06.324257 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.324260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.324321 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.324411 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:06.324795 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.325592 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.327183 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.327846 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.327947 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.328119 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.329262 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.329360 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5113}
I20260926 14:45:06.329957 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 LEARNER]: Refusing update from remote peer 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 14:45:06.330432 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 14:45:06.331676 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 42fa761df91a4ca98f4f47806f82fc65 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.333619 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60eabd2bd7fd43d7b3dfc3545f3fbbfc. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.333734 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:45:06.333804 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: true } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.333873 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:45:06.334174 7043 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: attempting to promote NON_VOTER 8a07669340f7465cba698cb64034b4a1 to VOTER
I20260926 14:45:06.334649 7043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } }
I20260926 14:45:06.336598 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 LEARNER]: Refusing update from remote peer 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 14:45:06.336849 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.336525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46531 (local address 127.0.0.1:51126)
0926 14:45:06.336702 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.336768 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260926 14:45:06.336966 6470 consensus_peers.cc:487] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc -> Peer 42fa761df91a4ca98f4f47806f82fc65 (127.0.98.60:46531): Couldn't send request to peer 42fa761df91a4ca98f4f47806f82fc65. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:46531: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 14:45:06.337168 7044 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 14:45:06.338824 7053 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8a07669340f7465cba698cb64034b4a1 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.339308 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8a07669340f7465cba698cb64034b4a1 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.340337 7053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.340440 7053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:06.340473 7043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.340575 7043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:06.341521 7238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.341637 7238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:45:06.347666 7055 master.cc:552] Master@127.0.98.60:46531 shutdown complete.
I20260926 14:45:06.347725 7055 master_runner.cc:305] Clearing existing system tablet
I20260926 14:45:06.347960 7055 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:06.348003 7055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:06.349350 7055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:06.350251 7240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.350570 7055 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:06.350700 7055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:06.350734 7055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/wal
uuid: "42fa761df91a4ca98f4f47806f82fc65"
format_stamp: "Formatted at 2026-09-26 14:45:06 on dist-test-slave-s7k6"
server_key: "143bfbb58e66c8c0ddcf5981dbc2c321"
server_key_iv: "b48171178690b78883f0d09aa7303b85"
server_key_version: "encryptionkey@0"
I20260926 14:45:06.352854 7055 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:06.355439 7055 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 14:45:06.355537 7055 log.cc:1196] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 14:45:06.355778 7055 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: Deleting consensus metadata
I20260926 14:45:06.355995 7055 master_runner.cc:315] Copying system tablet from 127.0.98.62:38541
I20260926 14:45:06.357192 7055 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:38541
I20260926 14:45:06.366634 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.357878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38794 (local address 127.0.98.62:38541)
0926 14:45:06.358010 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.358016 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.358209 (+ 193us) server_negotiation.cc:407] Connection header received
0926 14:45:06.358344 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.358347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.358400 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.358499 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.359065 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.359897 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.361136 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.361474 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.361581 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.363053 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.363070 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.363073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.363104 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.364781 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.365290 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.365294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.365296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.365386 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.365677 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.365680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.365682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.365908 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.365979 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.366344 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.366468 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:45:06.367199 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.357773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:38794)
0926 14:45:06.358104 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.358120 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.358248 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.358540 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.358543 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.358818 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.358936 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.358944 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.360028 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.360032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.360934 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.360947 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.361723 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.361727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.361827 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.362414 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.362429 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.362890 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.364907 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.364911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.364913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.365177 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.365493 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.365496 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.365498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.365580 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.366047 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.366050 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.366161 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.366880 (+ 719us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.367027 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":99,"threads_started":1}
I20260926 14:45:06.367717 6504 tablet_copy_service.cc:140] P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 42fa761df91a4ca98f4f47806f82fc65 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:38794)
I20260926 14:45:06.367823 6504 tablet_copy_service.cc:161] P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 42fa761df91a4ca98f4f47806f82fc65 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:38794: session id = 42fa761df91a4ca98f4f47806f82fc65-00000000000000000000000000000000
I20260926 14:45:06.368935 6504 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 14:45:06.370224 7055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 14:45:06.372805 7055 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: tablet copy: Starting download of 0 data blocks...
I20260926 14:45:06.373015 7055 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: tablet copy: Starting download of 1 WAL segments...
I20260926 14:45:06.374534 7055 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 14:45:06.376371 6504 tablet_copy_service.cc:342] P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Request end of tablet copy session 42fa761df91a4ca98f4f47806f82fc65-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:38794
I20260926 14:45:06.376464 6504 tablet_copy_service.cc:433] P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: ending tablet copy session 42fa761df91a4ca98f4f47806f82fc65-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 42fa761df91a4ca98f4f47806f82fc65
I20260926 14:45:06.384954 7055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:06.386860 7248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:06.386958 7250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:06.386874 7247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:06.387439 7055 server_base.cc:1034] running on GCE node
I20260926 14:45:06.387593 7055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:06.387897 7055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:06.389053 7055 hybrid_clock.cc:648] HybridClock initialized: now 1790433906389039 us; error 32 us; skew 500 ppm
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:06.390823 7055 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 14:45:06.391300 7055 webserver.cc:460] Webserver started at http://127.0.98.60:37345/ using document root <none> and password file <none>
I20260926 14:45:06.391477 7055 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:06.391522 7055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:06.392812 7055 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:06.393854 7257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.394130 7055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:06.394205 7055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/wal
uuid: "42fa761df91a4ca98f4f47806f82fc65"
format_stamp: "Formatted at 2026-09-26 14:45:06 on dist-test-slave-s7k6"
server_key: "143bfbb58e66c8c0ddcf5981dbc2c321"
server_key_iv: "b48171178690b78883f0d09aa7303b85"
server_key_version: "encryptionkey@0"
I20260926 14:45:06.394284 7055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:06.407677 7055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:06.409828 7055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:06.414820 7309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:46531 every 8 connection(s)
I20260926 14:45:06.414832 7055 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:46531
I20260926 14:45:06.415550 7055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
I20260926 14:45:06.417439 7310 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:45:06.418569 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: Bootstrap starting.
W20260926 14:45:06.424661 7235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:06.424921 7235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:06.424979 7235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:06.429240 7235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:06.429306 7235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:06.429389 7235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:45:06.430076 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:45:06.430365 7310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: Bootstrap complete.
I20260926 14:45:06.430998 7310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } }
I20260926 14:45:06.431155 7310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 42fa761df91a4ca98f4f47806f82fc65, State: Initialized, Role: LEARNER
I20260926 14:45:06.431260 7310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } }
I20260926 14:45:06.431706 7310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:06.431680 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.431897 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: This master's current role is: LEARNER
I20260926 14:45:06.436309 7327 catalog_manager.cc:1259] Loaded cluster ID: 97faa1410e51424e899ad395ff4fefd7
I20260926 14:45:06.436260 7235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38427
--tserver_master_addrs=127.0.98.62:38541,127.0.98.61:43045,127.0.98.60:46531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:06.436614 7327 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: loading cluster ID for follower catalog manager: success
I20260926 14:45:06.437625 7235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:06.438949 7235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:45:06.438951 7327 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: acquiring CA information for follower catalog manager: success
I20260926 14:45:06.440555 7327 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 14:45:06.448982 7330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:06.449000 7329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:06.449007 7332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:06.449345 7235 server_base.cc:1034] running on GCE node
I20260926 14:45:06.449771 7235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:06.450410 7235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:06.451556 7235 hybrid_clock.cc:648] HybridClock initialized: now 1790433906451526 us; error 44 us; skew 500 ppm
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:06.454226 7235 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 14:45:06.455192 7235 webserver.cc:460] Webserver started at http://127.0.98.1:38427/ using document root <none> and password file <none>
I20260926 14:45:06.455724 7235 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:06.455794 7235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:06.458743 6809 heartbeater.cc:503] Master 127.0.98.62:38541 was elected leader, sending a full tablet report...
I20260926 14:45:06.459748 7235 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:45:06.462018 7339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.463383 7235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:06.463496 7235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "082247f502db4b6d88447f749061b795"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "6e6729aa5ecc6c6e249443698f352035"
server_key_iv: "c317f559c452902c78efa5983e428836"
server_key_version: "encryptionkey@0"
I20260926 14:45:06.464051 7235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:06.485464 7235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:06.488862 7235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:06.489044 7235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:06.489666 7235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:06.490554 7235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:06.490609 7235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.490650 7235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:06.490680 7235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.500157 7235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35681
I20260926 14:45:06.500178 7452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35681 every 8 connection(s)
I20260926 14:45:06.501183 7235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:45:06.502909 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 7235
I20260926 14:45:06.503096 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 6680
I20260926 14:45:06.510828 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43973
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38541
--webserver_port=41577
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38541,127.0.98.61:43045
--tserver_master_addrs=127.0.98.62:38541,127.0.98.61:43045,127.0.98.60:46531 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 14:45:06.518496 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.503610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35413 (local address 127.0.98.62:38541)
0926 14:45:06.503768 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.503773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.504301 (+ 528us) server_negotiation.cc:407] Connection header received
0926 14:45:06.505439 (+ 1138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.505442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.505498 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.505583 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.507172 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.507922 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.509713 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.510063 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.510145 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.514010 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.514026 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.514029 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.514059 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.516066 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.516676 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.516680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.516682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.516875 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.517168 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.517171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.517173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.517390 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.517467 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.518190 (+ 723us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.518324 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:45:06.518496 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.504807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48879 (local address 127.0.98.61:43045)
0926 14:45:06.504928 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.504932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.507145 (+ 2213us) server_negotiation.cc:407] Connection header received
0926 14:45:06.507330 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.507333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.507389 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.507465 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.508026 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.508783 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.509901 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.510213 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.510282 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.514115 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.514118 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.514121 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.514146 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.516001 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.516676 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.516680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.516682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.516778 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.517183 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.517187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.517188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.517390 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.517467 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.518189 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.518328 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:45:06.519657 7457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.503613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.1:35413)
0926 14:45:06.504157 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.504209 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.505201 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.505770 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.505778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.506289 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.506994 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.507024 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.508087 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.508090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.509570 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.509582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.510353 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.510357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.510574 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.511549 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.511569 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.513802 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.516164 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.516166 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.516168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.516559 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.516960 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.516963 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.516964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.517070 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.517543 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.517546 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.517869 (+ 323us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.518659 (+ 790us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.518820 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":57,"spinlock_wait_cycles":24320,"thread_start_us":625,"threads_started":1}
I20260926 14:45:06.519996 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.503613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43045 (local address 127.0.98.1:48879)
0926 14:45:06.507052 (+ 3439us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.507066 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.507213 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.507576 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.507579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.507778 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.507893 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.507899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.508914 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.508917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.509772 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.509781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.510428 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.510432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.510570 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.512050 (+ 1480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.512067 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.513977 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.516161 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.516164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.516166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.516559 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.516987 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.516990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.516992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.517064 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.517643 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.517645 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.517869 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.518809 (+ 940us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.518923 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3345,"mutex_wait_us":147,"spinlock_wait_cycles":11008,"thread_start_us":106,"threads_started":1}
I20260926 14:45:06.520084 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.504975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58111 (local address 127.0.98.60:46531)
0926 14:45:06.507436 (+ 2461us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.507449 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.507824 (+ 375us) server_negotiation.cc:407] Connection header received
0926 14:45:06.507869 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.507872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.507918 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.507994 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.508548 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.509592 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.510810 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.511094 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.511183 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.514012 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.514016 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.514018 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.514046 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.515836 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.516579 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.516583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.516585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.516676 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.517087 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.517091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.517093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.517341 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.517439 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.519768 (+ 2329us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.519903 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2381,"thread_start_us":77,"threads_started":1}
I20260926 14:45:06.519666 7458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.504658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46531 (local address 127.0.98.1:58111)
0926 14:45:06.507649 (+ 2991us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.507662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.507744 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.508105 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.508108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.508297 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.508403 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.508409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.509699 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.509702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.510661 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.510669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.511216 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.511220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.511305 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.511707 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.511719 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.513801 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.515979 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.515997 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.516016 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.516459 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.516779 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.516782 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.516784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.516967 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.517495 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.517503 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.517878 (+ 375us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.518105 (+ 227us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.518495 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2919,"mutex_wait_us":3,"spinlock_wait_cycles":78592,"thread_start_us":71,"threads_started":1}
I20260926 14:45:06.521306 7455 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43045
I20260926 14:45:06.521327 7454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38541
I20260926 14:45:06.521378 7453 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46531
I20260926 14:45:06.521554 7455 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:06.521554 7453 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:06.521554 7454 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:06.522156 7455 heartbeater.cc:511] Master 127.0.98.61:43045 requested a full tablet report, sending...
I20260926 14:45:06.522156 7454 heartbeater.cc:511] Master 127.0.98.62:38541 requested a full tablet report, sending...
I20260926 14:45:06.522171 7453 heartbeater.cc:511] Master 127.0.98.60:46531 requested a full tablet report, sending...
I20260926 14:45:06.523674 6483 ts_manager.cc:194] Re-registered known tserver with Master: 082247f502db4b6d88447f749061b795 (127.0.98.1:35681)
I20260926 14:45:06.523892 7274 ts_manager.cc:194] Registered new tserver with Master: 082247f502db4b6d88447f749061b795 (127.0.98.1:35681)
I20260926 14:45:06.523895 7091 ts_manager.cc:194] Registered new tserver with Master: 082247f502db4b6d88447f749061b795 (127.0.98.1:35681)
I20260926 14:45:06.524410 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35413
W20260926 14:45:06.627020 7461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:06.627283 7461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:06.627341 7461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:06.631318 7461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:06.631377 7461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:06.631454 7461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:45:06.636981 7461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41577
--tserver_master_addrs=127.0.98.62:38541,127.0.98.61:43045,127.0.98.60:46531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:06.638094 7461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:06.639213 7461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:06.648878 7467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:06.648897 7466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:06.648878 7469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:06.648983 7461 server_base.cc:1034] running on GCE node
I20260926 14:45:06.649490 7461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:06.650146 7461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:06.651305 7461 hybrid_clock.cc:648] HybridClock initialized: now 1790433906651276 us; error 42 us; skew 500 ppm
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:06.653838 7461 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 14:45:06.654629 7461 webserver.cc:460] Webserver started at http://127.0.98.2:41577/ using document root <none> and password file <none>
I20260926 14:45:06.655169 7461 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:06.655247 7461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:06.659236 7461 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:45:06.661367 7476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.662686 7461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:45:06.662837 7461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "22f4ab545e034fdbb4cfa9a872941fd9"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "499d9b12fe9072e312adc037572efb9b"
server_key_iv: "4db54656d9b4b48f83dea24a32b43c1c"
server_key_version: "encryptionkey@0"
I20260926 14:45:06.663385 7461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:06.664287 6943 heartbeater.cc:503] Master 127.0.98.62:38541 was elected leader, sending a full tablet report...
I20260926 14:45:06.697355 7461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:06.700567 7461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:06.700762 7461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:06.701383 7461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:06.702251 7461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:06.702299 7461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.702353 7461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:06.702380 7461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.710816 7461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43973
I20260926 14:45:06.710844 7589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43973 every 8 connection(s)
I20260926 14:45:06.711835 7461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:45:06.719203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 7461
I20260926 14:45:06.719395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 6814
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 14:45:06.728997 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.713682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48931 (local address 127.0.98.61:43045)
0926 14:45:06.713836 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.713841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.715078 (+ 1237us) server_negotiation.cc:407] Connection header received
0926 14:45:06.716343 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.716347 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.716400 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.716484 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.717936 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.718741 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.720324 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.720680 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.720758 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.724903 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.724907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.724910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.724949 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.726526 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.727196 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.727200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.727202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.727289 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.727698 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.727701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.727703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.727935 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.728010 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.728686 (+ 676us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.728835 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 14:45:06.728998 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.713689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55137 (local address 127.0.98.62:38541)
0926 14:45:06.713936 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.713940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.714898 (+ 958us) server_negotiation.cc:407] Connection header received
0926 14:45:06.716067 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.716071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.716130 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.716205 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.717866 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.718718 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.720357 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.720686 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.720758 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.725236 (+ 4478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.725240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.725243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.725269 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.727048 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.727549 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.727553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.727555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.727651 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.727914 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.727917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.727918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.728104 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.728168 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.728713 (+ 545us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.728833 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260926 14:45:06.729038 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.715454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44689 (local address 127.0.98.60:46531)
0926 14:45:06.715575 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.715579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.715595 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:45:06.716653 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.716657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.716714 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.716793 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.717886 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.718484 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.720326 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.720730 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.720796 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.725740 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.725744 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.725747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.725778 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.727434 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.727856 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.727860 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.727862 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.727935 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.728229 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.728232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.728234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.728422 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.728490 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.728795 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.728914 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:45:06.729987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35215
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38541
--webserver_port=38399
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:38541,127.0.98.61:43045
--tserver_master_addrs=127.0.98.62:38541,127.0.98.61:43045,127.0.98.60:46531 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:45:06.730863 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.714111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.2:55137)
0926 14:45:06.714682 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.714743 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.715851 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.716420 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.716429 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.717062 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.717713 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.717729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.718883 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.718886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.720213 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.720224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.720964 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.720967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.721254 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.721981 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.722002 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.725088 (+ 3086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.727215 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.727218 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.727220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.727445 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.727751 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.727754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.727756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.727832 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.728232 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.728235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.728418 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.729191 (+ 773us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.729649 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":220,"spinlock_wait_cycles":186752,"thread_start_us":657,"threads_started":1}
I20260926 14:45:06.730858 7595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.714111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43045 (local address 127.0.98.2:48931)
0926 14:45:06.714987 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.714999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.715850 (+ 851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.716528 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.716531 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.717072 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.717721 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.717730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.718847 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.718851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.720215 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.720222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.720964 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.720967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.721248 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.722027 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.722043 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.724697 (+ 2654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.726664 (+ 1967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.726670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.726691 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.727080 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.727396 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.727399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.727401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.727594 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.728068 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.728074 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.728421 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.729197 (+ 776us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.729881 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"mutex_wait_us":600,"spinlock_wait_cycles":22528,"thread_start_us":133,"threads_started":1}
I20260926 14:45:06.730858 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.715108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46531 (local address 127.0.98.2:44689)
0926 14:45:06.715351 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.715364 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.715855 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.716839 (+ 984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.716842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.717105 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.717723 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.717730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.718632 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.718635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.720184 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.720195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.720964 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.720967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.721228 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.722116 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.722129 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.725087 (+ 2958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.727534 (+ 2447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.727537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.727538 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.727762 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.728039 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.728055 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.728057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.728136 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.728555 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.728557 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.728637 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.729636 (+ 999us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.729765 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":217,"spinlock_wait_cycles":24448,"thread_start_us":83,"threads_started":1}
I20260926 14:45:06.732177 7591 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38541
I20260926 14:45:06.732173 7592 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43045
I20260926 14:45:06.732204 7590 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46531
I20260926 14:45:06.732448 7591 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:06.732450 7590 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:06.732448 7592 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:06.733065 7591 heartbeater.cc:511] Master 127.0.98.62:38541 requested a full tablet report, sending...
I20260926 14:45:06.733054 7592 heartbeater.cc:511] Master 127.0.98.61:43045 requested a full tablet report, sending...
I20260926 14:45:06.733052 7590 heartbeater.cc:511] Master 127.0.98.60:46531 requested a full tablet report, sending...
I20260926 14:45:06.734169 6483 ts_manager.cc:194] Re-registered known tserver with Master: 22f4ab545e034fdbb4cfa9a872941fd9 (127.0.98.2:43973)
I20260926 14:45:06.734237 7274 ts_manager.cc:194] Registered new tserver with Master: 22f4ab545e034fdbb4cfa9a872941fd9 (127.0.98.2:43973)
I20260926 14:45:06.734370 7091 ts_manager.cc:194] Registered new tserver with Master: 22f4ab545e034fdbb4cfa9a872941fd9 (127.0.98.2:43973)
I20260926 14:45:06.734941 6483 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55137
I20260926 14:45:06.737571 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.724303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.0.98.60:46531)
0926 14:45:06.724586 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.724591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.724946 (+ 355us) server_negotiation.cc:407] Connection header received
0926 14:45:06.726092 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.726096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.726161 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.729993 (+ 3832us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:06.731969 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.735456 (+ 3487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.736443 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.737063 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.737158 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.737305 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.737369 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.737438 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":108,"threads_started":1}
I20260926 14:45:06.737568 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.724084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46531 (local address 127.0.0.1:51138)
0926 14:45:06.724567 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.724582 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.724707 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.728975 (+ 4268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.728979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.728997 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:06.729086 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.729093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.735572 (+ 6479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.735576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.736321 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.736331 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.737173 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.737177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.737266 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.737287 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.737360 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.737439 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390}
I20260926 14:45:06.738219 7284 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 14:45:06.738579 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60eabd2bd7fd43d7b3dfc3545f3fbbfc. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.738724 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: This master's current role is: LEARNER
W20260926 14:45:06.844255 7598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:06.844520 7598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:06.844568 7598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:06.848878 7598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:06.848942 7598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:06.848999 7598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:45:06.854271 7598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35215
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38399
--tserver_master_addrs=127.0.98.62:38541,127.0.98.61:43045,127.0.98.60:46531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:06.855335 7598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:06.856285 7598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:06.865804 7604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:06.865815 7603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:06.865815 7606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:06.866180 7598 server_base.cc:1034] running on GCE node
I20260926 14:45:06.866600 7598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:06.867280 7598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:06.868448 7598 hybrid_clock.cc:648] HybridClock initialized: now 1790433906868426 us; error 30 us; skew 500 ppm
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:06.871268 7598 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 14:45:06.872228 7598 webserver.cc:460] Webserver started at http://127.0.98.3:38399/ using document root <none> and password file <none>
I20260926 14:45:06.872812 7598 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:06.872903 7598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:06.877022 7598 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:45:06.879305 7613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.880863 7598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:06.880983 7598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "a195a88278b243b3b6463201766eeb3d"
format_stamp: "Formatted at 2026-09-26 14:45:05 on dist-test-slave-s7k6"
server_key: "a3d270734c4cdb5a87c2d314db3b9c3b"
server_key_iv: "61ee2af32ad8043e403fc5dd71ee43b0"
server_key_version: "encryptionkey@0"
I20260926 14:45:06.881583 7598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:06.900810 7598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:06.903949 7598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:06.904101 7598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:06.904693 7598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:06.905547 7598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:06.905596 7598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.905637 7598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:06.905663 7598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:06.914582 7598 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35215
I20260926 14:45:06.914665 7726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35215 every 8 connection(s)
I20260926 14:45:06.915692 7598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:45:06.917874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 7598
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 14:45:06 dist-test-slave-s7k6 krb5kdc[6439](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433906, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 14:45:06.931049 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.920611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.0.98.60:46531)
0926 14:45:06.920724 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.920728 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.921794 (+ 1066us) server_negotiation.cc:407] Connection header received
0926 14:45:06.921945 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.921948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.921988 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.922035 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.922944 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.923633 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.924778 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.925105 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.925186 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.927248 (+ 2062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.927251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.927254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.927279 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.928916 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.929607 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.929611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.929613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.929708 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.930006 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.930010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.930011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.930256 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.930362 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.930728 (+ 366us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.930880 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:45:06.931779 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.920056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.0.98.62:38541)
0926 14:45:06.920280 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.920284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.920822 (+ 538us) server_negotiation.cc:407] Connection header received
0926 14:45:06.921008 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.921012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.921058 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.921118 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.922147 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.923023 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.924181 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.924534 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.924608 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.926633 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.926637 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.926639 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.926996 (+ 357us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.929030 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.929944 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.929948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.929950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.930052 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.930429 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.930432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.930434 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.930683 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.930813 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.931460 (+ 647us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.931604 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20260926 14:45:06.934825 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.917412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45427 (local address 127.0.98.60:46531)
0926 14:45:06.917542 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.917614 (+ 72us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.919096 (+ 1482us) server_negotiation.cc:407] Connection header received
0926 14:45:06.921208 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.921212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.921267 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.921323 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.922402 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.923208 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.925683 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.925994 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.926054 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.930299 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.930303 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.930305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.930331 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.932322 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.933018 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.933021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.933023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.933118 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.933548 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.933552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.933554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.933760 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.933831 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.934524 (+ 693us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.934659 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:45:06.934831 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.917557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49693 (local address 127.0.98.61:43045)
0926 14:45:06.919713 (+ 2156us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.919718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.919883 (+ 165us) server_negotiation.cc:407] Connection header received
0926 14:45:06.920364 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.920368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.920424 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.920475 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.923785 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.924524 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.925583 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.925900 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.925973 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.930784 (+ 4811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.930788 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.930790 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.930813 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.932661 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.933187 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.933191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.933193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.933276 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.933615 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.933619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.933621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.933836 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.933916 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.934530 (+ 614us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.934643 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2067}
I20260926 14:45:06.936399 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.917965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46531 (local address 127.0.98.3:45427)
0926 14:45:06.918909 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.918980 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.920105 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.921465 (+ 1360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.921468 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.921682 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.922249 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.922264 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.923458 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.923461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.924771 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.924780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.926190 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.926193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.926410 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.927118 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.927131 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.929704 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.932492 (+ 2788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.932498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.932514 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.932834 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.933225 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.933227 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.933229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.933406 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.933908 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.933914 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.934248 (+ 334us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.934991 (+ 743us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.935376 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":61,"spinlock_wait_cycles":26496,"thread_start_us":694,"threads_started":1}
I20260926 14:45:06.936425 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.919500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33101 (local address 127.0.98.62:38541)
0926 14:45:06.919627 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.919632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.919925 (+ 293us) server_negotiation.cc:407] Connection header received
0926 14:45:06.920424 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.920428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.920481 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.920533 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.922399 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.923123 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.925638 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.925958 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.926046 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.933006 (+ 6960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.933010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.933013 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.933039 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.934888 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.935402 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.935405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.935407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.935481 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.935725 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.935729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.935730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.935886 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.935963 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.936195 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.936292 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:45:06.936704 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.919433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.3:33101)
0926 14:45:06.919858 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.919868 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.920313 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.920916 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.920926 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.922088 (+ 1162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.922246 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.922264 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.923265 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.923268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.925509 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.925517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.926186 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.926190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.926890 (+ 700us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.927301 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.927312 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.930155 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.935052 (+ 4897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.935055 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.935057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.935300 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.935559 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.935562 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.935564 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.935640 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.935968 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.935971 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.936050 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.936129 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.936254 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":1813,"thread_start_us":75,"threads_started":1}
I20260926 14:45:06.937278 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.917965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43045 (local address 127.0.98.3:49693)
0926 14:45:06.919797 (+ 1832us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.919812 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.920110 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.920915 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.920926 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.921489 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:06.923649 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.923658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.924655 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.924657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.925454 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.925464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.926181 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.926185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.926407 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.927045 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:06.927075 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:06.930603 (+ 3528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:06.932792 (+ 2189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.932795 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.932797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.933066 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.933396 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:06.933399 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:06.933401 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:06.933485 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:06.933995 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:06.933998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:06.934252 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.934969 (+ 717us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.937118 (+ 2149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1599,"mutex_wait_us":3419,"spinlock_wait_cycles":6272,"thread_start_us":196,"threads_started":1}
I20260926 14:45:06.937902 7729 heartbeater.cc:347] Connected to a master server at 127.0.98.61:43045
I20260926 14:45:06.937886 7727 heartbeater.cc:347] Connected to a master server at 127.0.98.60:46531
I20260926 14:45:06.937886 7728 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38541
I20260926 14:45:06.938181 7728 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:06.938158 7727 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:06.938156 7729 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:06.935184 6483 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 14:45:06.938776 7729 heartbeater.cc:511] Master 127.0.98.61:43045 requested a full tablet report, sending...
I20260926 14:45:06.938895 7728 heartbeater.cc:511] Master 127.0.98.62:38541 requested a full tablet report, sending...
I20260926 14:45:06.938786 7727 heartbeater.cc:511] Master 127.0.98.60:46531 requested a full tablet report, sending...
W20260926 14:45:06.939828 6483 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:45:06.939896 7091 ts_manager.cc:194] Registered new tserver with Master: a195a88278b243b3b6463201766eeb3d (127.0.98.3:35215)
I20260926 14:45:06.939950 6484 ts_manager.cc:194] Re-registered known tserver with Master: a195a88278b243b3b6463201766eeb3d (127.0.98.3:35215)
I20260926 14:45:06.939949 7274 ts_manager.cc:194] Registered new tserver with Master: a195a88278b243b3b6463201766eeb3d (127.0.98.3:35215)
I20260926 14:45:06.940619 6484 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33101
I20260926 14:45:06.942792 7744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: true } } }
I20260926 14:45:06.942921 7744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:06.946426 7744 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: attempting to promote NON_VOTER 42fa761df91a4ca98f4f47806f82fc65 to VOTER
I20260926 14:45:06.946830 7744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } }
I20260926 14:45:06.947875 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 FOLLOWER]: Refusing update from remote peer 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 14:45:06.948246 7744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 14:45:06.948312 7284 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 LEARNER]: Refusing update from remote peer 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 14:45:06.948994 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.920985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.0.98.61:43045)
0926 14:45:06.921587 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.921591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.928813 (+ 7222us) server_negotiation.cc:407] Connection header received
0926 14:45:06.929012 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.929015 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.929066 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.929202 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:06.941620 (+ 12418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.942159 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.943289 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.943523 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.943605 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.945221 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:06.945225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:06.945228 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:06.945254 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:06.946940 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.947414 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.947417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.947419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.947506 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:06.947791 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:06.947794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:06.947795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:06.948012 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:06.948126 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.948712 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.948833 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":73,"threads_started":1}
I20260926 14:45:06.949455 7744 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260926 14:45:06.950006 7044 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 42fa761df91a4ca98f4f47806f82fc65 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:06.950315 7101 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 42fa761df91a4ca98f4f47806f82fc65 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:06.951459 7745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:06.951480 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:06.951581 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:06.951668 7044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:06.951737 7044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:06.951570 7745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:06.952244 7284 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 42fa761df91a4ca98f4f47806f82fc65 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:06.953800 7316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:06.953913 7316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:45:06.954589 7754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:06.954689 7754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:45:06.967706 7041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.961947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35215 (local address 127.0.0.1:45568)
0926 14:45:06.962263 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.962278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.962357 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.963725 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.963728 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.963746 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:06.963858 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.963865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.965076 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.965079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.966135 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.966148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.967328 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.967331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.967406 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.967430 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.967493 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.967566 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240}
I20260926 14:45:06.967779 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.962892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.0.98.3:35215)
0926 14:45:06.963223 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.963235 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.963260 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:45:06.963337 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.963343 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.963584 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.963749 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:06.963946 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.964965 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.966304 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.967209 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.967327 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.967478 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.967532 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.967630 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":115,"threads_started":1}
I20260926 14:45:06.970304 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.962567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43973 (local address 127.0.0.1:54282)
0926 14:45:06.964310 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.964323 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.964477 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.965098 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.965100 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.965117 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:06.965214 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.965220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.967607 (+ 2387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.967610 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.969108 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.969118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.969896 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.969899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.969993 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.970014 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.970090 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.970164 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1684,"thread_start_us":104,"threads_started":1}
I20260926 14:45:06.970371 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.963265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54282 (local address 127.0.98.2:43973)
0926 14:45:06.963646 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.963659 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.964409 (+ 750us) server_negotiation.cc:407] Connection header received
0926 14:45:06.964615 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.964621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.964816 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.964917 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:06.965328 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.967476 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.969222 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.969776 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.969882 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.970033 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.970119 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.970225 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":79,"threads_started":1}
I20260926 14:45:06.970745 7661 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf61f046a704b7ab14ad6d4a65a65b4 (DEFAULT_TABLE table=test-table [id=c69d1a29847e46749f5ea4d9bf342875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:06.971678 7756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.962412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35681 (local address 127.0.0.1:48030)
0926 14:45:06.962864 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:06.962876 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:06.962962 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:06.964568 (+ 1606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:06.964571 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:06.964586 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:06.964682 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.964688 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.966024 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.966027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.967161 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:06.967170 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.971273 (+ 4103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.971278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:06.971400 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.971423 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:45:06.971485 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 14:45:06.971557 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":118,"threads_started":1}
I20260926 14:45:06.972168 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:06.963597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.0.98.1:35681)
0926 14:45:06.964042 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:06.964058 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:06.964082 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:45:06.964159 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:06.964165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:06.964395 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:06.964900 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:06.965052 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.965933 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.967273 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:06.971113 (+ 3840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:06.971226 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:06.971405 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:06.971921 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 14:45:06.972030 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":119,"threads_started":1}
I20260926 14:45:06.972215 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf61f046a704b7ab14ad6d4a65a65b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:06.973430 7524 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf61f046a704b7ab14ad6d4a65a65b4 (DEFAULT_TABLE table=test-table [id=c69d1a29847e46749f5ea4d9bf342875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:06.974466 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf61f046a704b7ab14ad6d4a65a65b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:06.974910 7387 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf61f046a704b7ab14ad6d4a65a65b4 (DEFAULT_TABLE table=test-table [id=c69d1a29847e46749f5ea4d9bf342875]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:06.975872 7387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf61f046a704b7ab14ad6d4a65a65b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:06.979349 7761 tablet_bootstrap.cc:492] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: Bootstrap starting.
I20260926 14:45:06.981302 7761 tablet_bootstrap.cc:654] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:06.982188 7762 tablet_bootstrap.cc:492] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: Bootstrap starting.
I20260926 14:45:06.982486 7761 log.cc:824] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:06.983700 7764 tablet_bootstrap.cc:492] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: Bootstrap starting.
I20260926 14:45:06.984632 7762 tablet_bootstrap.cc:654] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:06.985343 7762 log.cc:824] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:06.985409 7764 tablet_bootstrap.cc:654] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:06.986016 7764 log.cc:824] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:06.987247 7762 tablet_bootstrap.cc:492] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: No bootstrap required, opened a new log
I20260926 14:45:06.987313 7761 tablet_bootstrap.cc:492] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: No bootstrap required, opened a new log
I20260926 14:45:06.987434 7762 ts_tablet_manager.cc:1397] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:45:06.987504 7761 ts_tablet_manager.cc:1397] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260926 14:45:06.987833 7764 tablet_bootstrap.cc:492] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: No bootstrap required, opened a new log
I20260926 14:45:06.987995 7764 ts_tablet_manager.cc:1397] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260926 14:45:06.991108 7762 raft_consensus.cc:348] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:06.991334 7762 raft_consensus.cc:374] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:06.991384 7762 raft_consensus.cc:729] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f4ab545e034fdbb4cfa9a872941fd9, State: Initialized, Role: FOLLOWER
I20260926 14:45:06.991796 7762 consensus_queue.cc:260] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:06.992058 7764 raft_consensus.cc:348] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:06.992051 7761 raft_consensus.cc:348] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:06.992311 7761 raft_consensus.cc:374] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:06.992311 7764 raft_consensus.cc:374] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:06.992368 7761 raft_consensus.cc:729] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a195a88278b243b3b6463201766eeb3d, State: Initialized, Role: FOLLOWER
I20260926 14:45:06.992376 7764 raft_consensus.cc:729] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082247f502db4b6d88447f749061b795, State: Initialized, Role: FOLLOWER
I20260926 14:45:06.992790 7762 ts_tablet_manager.cc:1428] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:45:06.992834 7764 consensus_queue.cc:260] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:06.992834 7761 consensus_queue.cc:260] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:06.993010 7591 heartbeater.cc:503] Master 127.0.98.62:38541 was elected leader, sending a full tablet report...
I20260926 14:45:06.993774 7764 ts_tablet_manager.cc:1428] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:45:06.993774 7761 ts_tablet_manager.cc:1428] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260926 14:45:06.994110 7728 heartbeater.cc:503] Master 127.0.98.62:38541 was elected leader, sending a full tablet report...
I20260926 14:45:06.996870 7454 heartbeater.cc:503] Master 127.0.98.62:38541 was elected leader, sending a full tablet report...
W20260926 14:45:07.002663 7456 tablet.cc:2369] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:45:07.166832 7730 tablet.cc:2369] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:45:07.188671 7767 raft_consensus.cc:482] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:45:07.188946 7767 raft_consensus.cc:504] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:07.190100 7767 leader_election.cc:288] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a195a88278b243b3b6463201766eeb3d (127.0.98.3:35215), 082247f502db4b6d88447f749061b795 (127.0.98.1:35681)
I20260926 14:45:07.193935 7594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:07.190384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35681 (local address 127.0.98.2:52813)
0926 14:45:07.190540 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:07.190555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:07.190661 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:07.190994 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:07.190997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:07.191011 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:07.191115 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:07.191120 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.191803 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.191806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:07.192638 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:07.192647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.193517 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.193521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:07.193625 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:07.193678 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 14:45:07.193742 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:45:07.193797 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":16,"spinlock_wait_cycles":6656}
I20260926 14:45:07.194056 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:07.190348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35215 (local address 127.0.98.2:32951)
0926 14:45:07.190540 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:07.190555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:07.190662 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:07.190921 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:07.190924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:07.190939 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:07.191047 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:07.191052 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.191743 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.191746 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:07.192616 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:07.192627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.193699 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.193701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:07.193793 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:07.193818 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:45:07.193881 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:45:07.193946 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":22,"spinlock_wait_cycles":13952}
I20260926 14:45:07.194056 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:07.190561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52813 (local address 127.0.98.1:35681)
0926 14:45:07.190734 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:07.190739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:07.190779 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:45:07.190858 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:07.190860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:07.190902 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:07.190968 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:07.191221 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.191707 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.192775 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.193364 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.193501 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:07.193564 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:07.193781 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 14:45:07.193889 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:45:07.194478 7758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:07.190431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32951 (local address 127.0.98.3:35215)
0926 14:45:07.190568 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:07.190572 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:07.190594 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:45:07.190790 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:07.190793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:07.190842 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:07.190904 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:07.191142 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.191643 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.192755 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.193364 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.193501 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:07.193576 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:07.194298 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 14:45:07.194375 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 14:45:07.194605 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf61f046a704b7ab14ad6d4a65a65b4" candidate_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082247f502db4b6d88447f749061b795" is_pre_election: true
I20260926 14:45:07.194962 7404 raft_consensus.cc:2454] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22f4ab545e034fdbb4cfa9a872941fd9 in term 0.
I20260926 14:45:07.195024 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf61f046a704b7ab14ad6d4a65a65b4" candidate_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a195a88278b243b3b6463201766eeb3d" is_pre_election: true
I20260926 14:45:07.195322 7681 raft_consensus.cc:2454] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 22f4ab545e034fdbb4cfa9a872941fd9 in term 0.
I20260926 14:45:07.195405 7478 leader_election.cc:302] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082247f502db4b6d88447f749061b795, 22f4ab545e034fdbb4cfa9a872941fd9; no voters:
I20260926 14:45:07.195881 7767 raft_consensus.cc:2781] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:45:07.195969 7767 raft_consensus.cc:482] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:45:07.196014 7767 raft_consensus.cc:3037] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:07.197073 7767 raft_consensus.cc:504] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:07.197441 7767 leader_election.cc:288] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [CANDIDATE]: Term 1 election: Requested vote from peers a195a88278b243b3b6463201766eeb3d (127.0.98.3:35215), 082247f502db4b6d88447f749061b795 (127.0.98.1:35681)
I20260926 14:45:07.197829 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf61f046a704b7ab14ad6d4a65a65b4" candidate_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a195a88278b243b3b6463201766eeb3d"
I20260926 14:45:07.197846 7404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf61f046a704b7ab14ad6d4a65a65b4" candidate_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082247f502db4b6d88447f749061b795"
I20260926 14:45:07.197952 7681 raft_consensus.cc:3037] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:07.197974 7404 raft_consensus.cc:3037] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:07.199122 7681 raft_consensus.cc:2454] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22f4ab545e034fdbb4cfa9a872941fd9 in term 1.
I20260926 14:45:07.199170 7404 raft_consensus.cc:2454] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 22f4ab545e034fdbb4cfa9a872941fd9 in term 1.
I20260926 14:45:07.199453 7478 leader_election.cc:302] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082247f502db4b6d88447f749061b795, 22f4ab545e034fdbb4cfa9a872941fd9; no voters:
I20260926 14:45:07.199640 7767 raft_consensus.cc:2781] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:07.199815 7767 raft_consensus.cc:686] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 1 LEADER]: Becoming Leader. State: Replica: 22f4ab545e034fdbb4cfa9a872941fd9, State: Running, Role: LEADER
I20260926 14:45:07.200237 7767 consensus_queue.cc:237] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:07.203747 6484 catalog_manager.cc:5495] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 22f4ab545e034fdbb4cfa9a872941fd9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } health_report { overall_health: UNKNOWN } } }
W20260926 14:45:07.213732 7593 tablet.cc:2369] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:45:07.258951 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:07.255628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54286 (local address 127.0.98.2:43973)
0926 14:45:07.255767 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:07.255771 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:07.255785 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:45:07.255862 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:07.255864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:07.255920 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:07.256015 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:45:07.256301 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.256815 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.257637 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.257841 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.257911 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:07.258161 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:45:07.258592 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:45:07.258681 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:07.258726 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 14:45:07.258807 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 14:45:07.264886 7404 raft_consensus.cc:1261] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795 [term 1 FOLLOWER]: Refusing update from remote peer 22f4ab545e034fdbb4cfa9a872941fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:07.264887 7681 raft_consensus.cc:1261] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d [term 1 FOLLOWER]: Refusing update from remote peer 22f4ab545e034fdbb4cfa9a872941fd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:07.265617 7770 consensus_queue.cc:1035] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:07.265803 7767 consensus_queue.cc:1035] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:07.278602 7775 mvcc.cc:205] Tried to move back new op lower bound from 7333617284150009856 to 7333617283895070720. Current Snapshot: MvccSnapshot[applied={T|T < 7333617284150009856}]
I20260926 14:45:07.283771 6483 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:07.283943 6483 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:07.288196 6483 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60eabd2bd7fd43d7b3dfc3545f3fbbfc: Sending DeleteTablet for 3 replicas of tablet 5bf61f046a704b7ab14ad6d4a65a65b4
I20260926 14:45:07.288969 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf61f046a704b7ab14ad6d4a65a65b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45568
I20260926 14:45:07.289132 7387 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf61f046a704b7ab14ad6d4a65a65b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48030
I20260926 14:45:07.289839 7783 tablet_replica.cc:330] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: stopping tablet replica
I20260926 14:45:07.289789 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bf61f046a704b7ab14ad6d4a65a65b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:07 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54282
I20260926 14:45:07.290247 7785 tablet_replica.cc:330] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: stopping tablet replica
I20260926 14:45:07.290257 7783 raft_consensus.cc:2229] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:07.290529 7785 raft_consensus.cc:2229] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:45:07.290712 7783 raft_consensus.cc:2258] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:07.290918 7785 raft_consensus.cc:2258] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:07.291285 7784 tablet_replica.cc:330] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: stopping tablet replica
I20260926 14:45:07.291602 7784 raft_consensus.cc:2229] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:07.291932 7784 raft_consensus.cc:2258] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:07.292237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 6452
I20260926 14:45:07.293140 7783 ts_tablet_manager.cc:1905] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:07.293527 7784 ts_tablet_manager.cc:1905] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:07.296317 7784 ts_tablet_manager.cc:1918] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:07.296479 7784 log.cc:1196] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-0/wal/wals/5bf61f046a704b7ab14ad6d4a65a65b4
I20260926 14:45:07.296835 7784 ts_tablet_manager.cc:1939] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 082247f502db4b6d88447f749061b795: Deleting consensus metadata
I20260926 14:45:07.296937 7783 ts_tablet_manager.cc:1918] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:07.297022 7783 log.cc:1196] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/5bf61f046a704b7ab14ad6d4a65a65b4
I20260926 14:45:07.297298 7783 ts_tablet_manager.cc:1939] T 5bf61f046a704b7ab14ad6d4a65a65b4 P a195a88278b243b3b6463201766eeb3d: Deleting consensus metadata
I20260926 14:45:07.298394 7785 ts_tablet_manager.cc:1905] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 14:45:07.299820 7340 connection.cc:535] server connection from 127.0.0.1:48030 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 14:45:07.300112 7617 connection.cc:535] server connection from 127.0.0.1:45568 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:45:07.301710 7785 ts_tablet_manager.cc:1918] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:07.301800 7785 log.cc:1196] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/5bf61f046a704b7ab14ad6d4a65a65b4
I20260926 14:45:07.302078 7785 ts_tablet_manager.cc:1939] T 5bf61f046a704b7ab14ad6d4a65a65b4 P 22f4ab545e034fdbb4cfa9a872941fd9: Deleting consensus metadata
W20260926 14:45:07.303036 7478 connection.cc:441] server connection from 127.0.0.1:54282 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54282 (request call id 1) could send its response
I20260926 14:45:07.310709 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:07.291049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.0.98.61:43045)
0926 14:45:07.291170 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:07.291174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:07.300759 (+ 9585us) server_negotiation.cc:407] Connection header received
0926 14:45:07.300946 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:07.300949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:07.301001 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:07.301048 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:07.302035 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.302888 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.304105 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.304461 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.304538 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:07.306292 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:07.306296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:07.306299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:07.306325 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:07.308038 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:07.309142 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:07.309146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:07.309148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:07.309236 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:07.309771 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:07.309775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:07.309777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:07.309988 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:07.310088 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:07.310422 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 14:45:07.310543 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:45:07.313275 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:07.303569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.0.98.61:43045)
0926 14:45:07.303677 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:07.303681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:07.304395 (+ 714us) server_negotiation.cc:407] Connection header received
0926 14:45:07.304533 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:07.304536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:07.304579 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:07.304625 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:07.305395 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.306108 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.307252 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.307579 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.307651 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:07.309232 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:07.309235 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:07.309238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:07.309265 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:07.311295 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:07.311986 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:07.311990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:07.311991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:07.312071 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:07.312366 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:07.312370 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:07.312371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:07.312570 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:07.312668 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:07.313005 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 14:45:07.313125 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:45:07.313931 7460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:07.303569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51174 (local address 127.0.98.60:46531)
0926 14:45:07.303677 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:07.303681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:07.304203 (+ 522us) server_negotiation.cc:407] Connection header received
0926 14:45:07.304380 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:07.304384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:07.304434 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:07.304484 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:07.305830 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.306550 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.307763 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.308091 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.308153 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:07.309640 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:07.309644 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:07.309646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:07.309674 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:07.312160 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:07.312556 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:07.312560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:07.312562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:07.312661 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:07.312948 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:07.312951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:07.312953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:07.313147 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:07.313237 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:07.313685 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 14:45:07.313784 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:45:07.319128 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:07.291049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.0.98.60:46531)
0926 14:45:07.291165 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:07.291169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:07.310800 (+ 19631us) server_negotiation.cc:407] Connection header received
0926 14:45:07.311512 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:07.311515 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:07.311560 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:07.311606 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:07.312386 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.313205 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.314373 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:07.314668 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:07.314815 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:07.316290 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:07.316294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:07.316297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:07.316321 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:07.317579 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:07.317958 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:07.317961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:07.317963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:07.318033 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:07.318289 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:07.318292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:07.318294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:07.318480 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:07.318600 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:07.318837 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 14:45:07.318952 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:45:08.278399 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.277877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.3:51811)
0926 14:45:08.278194 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.278284 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":76,"threads_started":1}
W20260926 14:45:08.278729 7728 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
I20260926 14:45:08.279263 7795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.278646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.1:57431)
0926 14:45:08.279073 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.279164 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":76,"threads_started":1}
W20260926 14:45:08.279795 7454 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
I20260926 14:45:08.285027 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.284498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.2:48243)
0926 14:45:08.284849 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.284928 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":68,"threads_started":1}
W20260926 14:45:08.285307 7591 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38541 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
I20260926 14:45:08.788198 7797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60eabd2bd7fd43d7b3dfc3545f3fbbfc)
I20260926 14:45:08.788434 7797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } }
I20260926 14:45:08.789484 7797 leader_election.cc:288] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60eabd2bd7fd43d7b3dfc3545f3fbbfc (127.0.98.62:38541), 8a07669340f7465cba698cb64034b4a1 (127.0.98.61:43045)
I20260926 14:45:08.789642 7798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 60eabd2bd7fd43d7b3dfc3545f3fbbfc)
I20260926 14:45:08.790055 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.789651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:39322)
0926 14:45:08.789890 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.789968 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":61,"threads_started":1}
I20260926 14:45:08.789992 7798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } }
W20260926 14:45:08.790346 7260 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
I20260926 14:45:08.791121 7798 leader_election.cc:288] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 60eabd2bd7fd43d7b3dfc3545f3fbbfc (127.0.98.62:38541), 42fa761df91a4ca98f4f47806f82fc65 (127.0.98.60:46531)
I20260926 14:45:08.791256 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.791016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:39334)
0926 14:45:08.791119 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.791181 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:45:08.791384 7260 leader_election.cc:334] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 60eabd2bd7fd43d7b3dfc3545f3fbbfc (127.0.98.62:38541): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
I20260926 14:45:08.791810 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.791425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:39342)
0926 14:45:08.791644 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.791716 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":74,"threads_started":1}
W20260926 14:45:08.792048 7077 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
I20260926 14:45:08.792824 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.792623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:39354)
0926 14:45:08.792716 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.792763 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 14:45:08.792995 7077 leader_election.cc:334] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 60eabd2bd7fd43d7b3dfc3545f3fbbfc (127.0.98.62:38541): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
I20260926 14:45:08.794570 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.789755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43045 (local address 127.0.0.1:38040)
0926 14:45:08.790187 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.790201 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:08.790313 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:08.790619 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:08.790622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:08.790644 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:08.790751 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.790784 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.792059 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.792062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.793342 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.793353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.794089 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.794093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.794211 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.794239 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 14:45:08.794321 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 14:45:08.794433 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":81,"threads_started":1}
I20260926 14:45:08.794582 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.790031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.0.98.61:43045)
0926 14:45:08.790359 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:08.790364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:08.790403 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:45:08.790455 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:08.790458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:08.790505 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:08.790589 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:08.790903 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.791922 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.793490 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.793951 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.794057 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.794123 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:08.794401 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 14:45:08.794473 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":66,"threads_started":1}
I20260926 14:45:08.795193 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42fa761df91a4ca98f4f47806f82fc65" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8a07669340f7465cba698cb64034b4a1" is_pre_election: true
I20260926 14:45:08.795430 7101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42fa761df91a4ca98f4f47806f82fc65 in term 1.
I20260926 14:45:08.795847 7259 leader_election.cc:302] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42fa761df91a4ca98f4f47806f82fc65, 8a07669340f7465cba698cb64034b4a1; no voters: 60eabd2bd7fd43d7b3dfc3545f3fbbfc
I20260926 14:45:08.795948 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.791762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:46531 (local address 127.0.0.1:51186)
0926 14:45:08.792022 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.792034 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:08.792123 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:08.792502 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:08.792505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:08.792523 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:08.792606 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.792612 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.793593 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.793596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.794783 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.794795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.795590 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.795594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.795685 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.795711 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:45:08.795779 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:45:08.795849 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":58,"threads_started":1}
I20260926 14:45:08.795995 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.791901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51186 (local address 127.0.98.60:46531)
0926 14:45:08.792129 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:08.792136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:08.792148 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:45:08.792271 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:08.792274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:08.792317 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:08.792393 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:08.792732 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.793482 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.794900 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.795454 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.795554 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.795618 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:08.795797 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 14:45:08.795873 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
I20260926 14:45:08.796187 7797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 14:45:08.796259 7797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 60eabd2bd7fd43d7b3dfc3545f3fbbfc)
I20260926 14:45:08.796339 7797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:45:08.797353 7797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } }
I20260926 14:45:08.797724 7797 leader_election.cc:288] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [CANDIDATE]: Term 2 election: Requested vote from peers 60eabd2bd7fd43d7b3dfc3545f3fbbfc (127.0.98.62:38541), 8a07669340f7465cba698cb64034b4a1 (127.0.98.61:43045)
I20260926 14:45:08.798074 7101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "42fa761df91a4ca98f4f47806f82fc65" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8a07669340f7465cba698cb64034b4a1"
I20260926 14:45:08.798148 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.797930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:39366)
0926 14:45:08.798025 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.798079 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 14:45:08.798192 7101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:45:08.798267 7284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a07669340f7465cba698cb64034b4a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "42fa761df91a4ca98f4f47806f82fc65" is_pre_election: true
I20260926 14:45:08.798493 7284 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8a07669340f7465cba698cb64034b4a1 in current term 2: Already voted for candidate 42fa761df91a4ca98f4f47806f82fc65 in this term.
I20260926 14:45:08.798856 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.798573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:39370)
0926 14:45:08.798710 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.798787 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 14:45:08.798929 7077 leader_election.cc:302] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8a07669340f7465cba698cb64034b4a1; no voters: 42fa761df91a4ca98f4f47806f82fc65, 60eabd2bd7fd43d7b3dfc3545f3fbbfc
W20260926 14:45:08.799022 7260 leader_election.cc:334] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 60eabd2bd7fd43d7b3dfc3545f3fbbfc (127.0.98.62:38541): Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
I20260926 14:45:08.799508 7101 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42fa761df91a4ca98f4f47806f82fc65 in term 2.
I20260926 14:45:08.799609 7798 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:45:08.799775 7259 leader_election.cc:302] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 42fa761df91a4ca98f4f47806f82fc65, 8a07669340f7465cba698cb64034b4a1; no voters: 60eabd2bd7fd43d7b3dfc3545f3fbbfc
I20260926 14:45:08.799979 7797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:45:08.800185 7797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [term 2 LEADER]: Becoming Leader. State: Replica: 42fa761df91a4ca98f4f47806f82fc65, State: Running, Role: LEADER
I20260926 14:45:08.800434 7797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } }
I20260926 14:45:08.801931 7807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42fa761df91a4ca98f4f47806f82fc65. Latest consensus state: current_term: 2 leader_uuid: "42fa761df91a4ca98f4f47806f82fc65" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:08.802042 7807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:08.802320 7809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:08.804498 7809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:08.804970 7809 catalog_manager.cc:1259] Loaded cluster ID: 97faa1410e51424e899ad395ff4fefd7
I20260926 14:45:08.805019 7809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:08.805567 7809 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:08.805949 7809 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: Loaded TSK: 0
I20260926 14:45:08.806643 7809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:45:08.825742 7274 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51174:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:45:08.828912 7274 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:45:08.832065 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.831804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:39376)
0926 14:45:08.831921 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.831980 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:45:08.832161 7101 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [term 2 FOLLOWER]: Refusing update from remote peer 42fa761df91a4ca98f4f47806f82fc65: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 14:45:08.832679 7797 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 14:45:08.832995 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.832735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.0.1:39380)
0926 14:45:08.832828 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.832930 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 14:45:08.833127 7260 consensus_peers.cc:487] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 -> Peer 60eabd2bd7fd43d7b3dfc3545f3fbbfc (127.0.98.62:38541): Couldn't send request to peer 60eabd2bd7fd43d7b3dfc3545f3fbbfc. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 14:45:08.834117 7798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42fa761df91a4ca98f4f47806f82fc65. Latest consensus state: current_term: 2 leader_uuid: "42fa761df91a4ca98f4f47806f82fc65" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:08.834233 7798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:45:08.835355 7797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "42fa761df91a4ca98f4f47806f82fc65" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:08.835487 7797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:08.835868 7798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "42fa761df91a4ca98f4f47806f82fc65" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "60eabd2bd7fd43d7b3dfc3545f3fbbfc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38541 } } peers { permanent_uuid: "8a07669340f7465cba698cb64034b4a1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43045 } attrs { promote: false } } peers { permanent_uuid: "42fa761df91a4ca98f4f47806f82fc65" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 46531 } attrs { promote: false } } }
I20260926 14:45:08.835994 7798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a07669340f7465cba698cb64034b4a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:45:08.846427 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.841961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35215 (local address 127.0.0.1:45580)
0926 14:45:08.842121 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.842134 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:08.842200 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:08.842891 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:08.842894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:08.842912 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:08.843024 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.843030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.844061 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.844064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.845113 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.845124 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.846029 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.846033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.846114 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.846143 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 14:45:08.846186 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 14:45:08.846264 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":24}
I20260926 14:45:08.846976 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.842346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.0.98.3:35215)
0926 14:45:08.842612 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:08.842617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:08.842633 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:45:08.842688 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:08.842691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:08.842730 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:08.843121 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:08.843222 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.843928 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.845243 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.845893 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.846633 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.846700 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:08.846739 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 14:45:08.846844 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260926 14:45:08.847105 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.842406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35681 (local address 127.0.0.1:48042)
0926 14:45:08.842555 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.842568 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:08.842642 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:08.843945 (+ 1303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:08.843948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:08.843966 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:08.844063 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.844069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.844823 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.844826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.845931 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.845940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.846723 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.846727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.846863 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.846893 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 14:45:08.846935 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 14:45:08.846988 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"mutex_wait_us":44}
I20260926 14:45:08.847357 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.842655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43973 (local address 127.0.0.1:54292)
0926 14:45:08.842927 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.842939 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:08.843007 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:08.843577 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:08.843580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:08.843598 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:08.843700 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.843705 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.844695 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.844698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.845946 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.845955 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.846953 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.846957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.847058 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.847083 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:45:08.847153 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 14:45:08.847226 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":77,"thread_start_us":108,"threads_started":1}
I20260926 14:45:08.847383 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.843094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.0.98.2:43973)
0926 14:45:08.843345 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:08.843350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:08.843364 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:45:08.843425 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:08.843429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:08.843473 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:08.843541 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:08.843822 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.844578 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.846096 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.846687 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.846826 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.846896 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:08.847172 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 14:45:08.847253 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260926 14:45:08.847586 7661 tablet_service.cc:1467] Processing CreateTablet for tablet be0edee4c8214ed4aa95cc91545e7634 (DEFAULT_TABLE table=test-table [id=d3d74c37314a4ed49c41c8160025ba12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:08.847654 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.842516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.0.98.1:35681)
0926 14:45:08.843736 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:08.843740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:08.843751 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:45:08.843799 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:08.843802 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:08.843847 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:08.843933 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:08.844159 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.844700 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.846071 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.846585 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.847351 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.847417 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:08.847452 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 14:45:08.847529 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":79,"threads_started":1}
I20260926 14:45:08.847884 7661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0edee4c8214ed4aa95cc91545e7634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:08.848192 7387 tablet_service.cc:1467] Processing CreateTablet for tablet be0edee4c8214ed4aa95cc91545e7634 (DEFAULT_TABLE table=test-table [id=d3d74c37314a4ed49c41c8160025ba12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:08.848434 7387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0edee4c8214ed4aa95cc91545e7634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:08.850529 7820 tablet_bootstrap.cc:492] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d: Bootstrap starting.
I20260926 14:45:08.850715 7821 tablet_bootstrap.cc:492] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795: Bootstrap starting.
I20260926 14:45:08.851661 7820 tablet_bootstrap.cc:654] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:08.851661 7821 tablet_bootstrap.cc:654] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:08.851984 7524 tablet_service.cc:1467] Processing CreateTablet for tablet be0edee4c8214ed4aa95cc91545e7634 (DEFAULT_TABLE table=test-table [id=d3d74c37314a4ed49c41c8160025ba12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:08.852210 7524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be0edee4c8214ed4aa95cc91545e7634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:08.852849 7820 tablet_bootstrap.cc:492] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d: No bootstrap required, opened a new log
I20260926 14:45:08.852943 7820 ts_tablet_manager.cc:1397] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:45:08.853132 7821 tablet_bootstrap.cc:492] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795: No bootstrap required, opened a new log
I20260926 14:45:08.853205 7821 ts_tablet_manager.cc:1397] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 14:45:08.853374 7820 raft_consensus.cc:348] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:08.853482 7820 raft_consensus.cc:374] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:08.853513 7820 raft_consensus.cc:729] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a195a88278b243b3b6463201766eeb3d, State: Initialized, Role: FOLLOWER
I20260926 14:45:08.853637 7820 consensus_queue.cc:260] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:08.853632 7821 raft_consensus.cc:348] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:08.853721 7821 raft_consensus.cc:374] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:08.853752 7821 raft_consensus.cc:729] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082247f502db4b6d88447f749061b795, State: Initialized, Role: FOLLOWER
I20260926 14:45:08.853880 7821 consensus_queue.cc:260] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:08.854030 7820 ts_tablet_manager.cc:1428] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:08.854182 7821 ts_tablet_manager.cc:1428] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:08.855715 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.855294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.3:36473)
0926 14:45:08.855544 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.855605 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":103,"threads_started":1}
I20260926 14:45:08.856441 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.856047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.1:51431)
0926 14:45:08.856267 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.856330 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
I20260926 14:45:08.856483 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.856245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.3:46971)
0926 14:45:08.856370 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.856417 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 14:45:08.856762 7727 heartbeater.cc:503] Master 127.0.98.60:46531 was elected leader, sending a full tablet report...
I20260926 14:45:08.857427 7826 tablet_bootstrap.cc:492] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9: Bootstrap starting.
I20260926 14:45:08.858353 7826 tablet_bootstrap.cc:654] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:08.858966 7453 heartbeater.cc:503] Master 127.0.98.60:46531 was elected leader, sending a full tablet report...
I20260926 14:45:08.859589 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.859348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.1:36679)
0926 14:45:08.859471 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.859525 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 14:45:08.859622 7826 tablet_bootstrap.cc:492] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9: No bootstrap required, opened a new log
I20260926 14:45:08.859697 7826 ts_tablet_manager.cc:1397] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:45:08.860107 7826 raft_consensus.cc:348] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:08.860270 7826 raft_consensus.cc:374] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:08.860314 7826 raft_consensus.cc:729] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22f4ab545e034fdbb4cfa9a872941fd9, State: Initialized, Role: FOLLOWER
I20260926 14:45:08.860453 7826 consensus_queue.cc:260] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:08.860765 7826 ts_tablet_manager.cc:1428] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:08.861490 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.861063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.2:54417)
0926 14:45:08.861330 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.861399 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":110,"threads_started":1}
I20260926 14:45:08.862138 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.861901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.2:47895)
0926 14:45:08.862032 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.862079 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260926 14:45:08.862258 7590 heartbeater.cc:503] Master 127.0.98.60:46531 was elected leader, sending a full tablet report...
I20260926 14:45:08.962946 7825 raft_consensus.cc:482] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:45:08.963119 7825 raft_consensus.cc:504] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:08.964309 7825 leader_election.cc:288] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a195a88278b243b3b6463201766eeb3d (127.0.98.3:35215), 22f4ab545e034fdbb4cfa9a872941fd9 (127.0.98.2:43973)
I20260926 14:45:08.968588 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.964650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57173 (local address 127.0.98.3:35215)
0926 14:45:08.964773 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:08.964778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:08.965010 (+ 232us) server_negotiation.cc:407] Connection header received
0926 14:45:08.965129 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:08.965132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:08.965189 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:08.965276 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:08.965681 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.966197 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.967474 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.967967 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.968061 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.968127 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:08.968387 (+ 260us) server_negotiation.cc:299] Negotiation successful
0926 14:45:08.968461 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:45:08.968636 7832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.964572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35215 (local address 127.0.98.1:57173)
0926 14:45:08.964877 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.964892 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:08.965034 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:08.965305 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:08.965308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:08.965333 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:08.965554 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.965563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.966318 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.966321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.967331 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.967344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.968110 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.968113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.968258 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.968296 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 14:45:08.968361 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:45:08.968448 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":79,"spinlock_wait_cycles":15744,"thread_start_us":126,"threads_started":1}
I20260926 14:45:08.968659 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.964651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34729 (local address 127.0.98.2:43973)
0926 14:45:08.964774 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:08.964778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:08.965016 (+ 238us) server_negotiation.cc:407] Connection header received
0926 14:45:08.965129 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:08.965132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:08.965189 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:08.965275 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:08.965675 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.966195 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.967515 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.968007 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.968102 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.968166 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:08.968455 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 14:45:08.968530 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:45:08.968834 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.964572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43973 (local address 127.0.98.1:34729)
0926 14:45:08.964877 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.964929 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:08.965034 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:08.965305 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:08.965308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:08.965333 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:08.965555 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.965563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.966313 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.966316 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.967390 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:08.967401 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:08.968116 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:08.968120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:08.968298 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:08.968322 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:45:08.968436 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 14:45:08.968509 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":134,"spinlock_wait_cycles":28416}
I20260926 14:45:08.969046 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0edee4c8214ed4aa95cc91545e7634" candidate_uuid: "082247f502db4b6d88447f749061b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a195a88278b243b3b6463201766eeb3d" is_pre_election: true
I20260926 14:45:08.969192 7681 raft_consensus.cc:2454] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082247f502db4b6d88447f749061b795 in term 0.
I20260926 14:45:08.969417 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0edee4c8214ed4aa95cc91545e7634" candidate_uuid: "082247f502db4b6d88447f749061b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" is_pre_election: true
I20260926 14:45:08.969583 7342 leader_election.cc:302] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082247f502db4b6d88447f749061b795, a195a88278b243b3b6463201766eeb3d; no voters:
I20260926 14:45:08.969677 7544 raft_consensus.cc:2454] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 082247f502db4b6d88447f749061b795 in term 0.
I20260926 14:45:08.969918 7825 raft_consensus.cc:2781] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:45:08.969987 7825 raft_consensus.cc:482] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:45:08.970031 7825 raft_consensus.cc:3037] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:08.971200 7825 raft_consensus.cc:504] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:08.971534 7825 leader_election.cc:288] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [CANDIDATE]: Term 1 election: Requested vote from peers a195a88278b243b3b6463201766eeb3d (127.0.98.3:35215), 22f4ab545e034fdbb4cfa9a872941fd9 (127.0.98.2:43973)
I20260926 14:45:08.971860 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0edee4c8214ed4aa95cc91545e7634" candidate_uuid: "082247f502db4b6d88447f749061b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a195a88278b243b3b6463201766eeb3d"
I20260926 14:45:08.971873 7544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be0edee4c8214ed4aa95cc91545e7634" candidate_uuid: "082247f502db4b6d88447f749061b795" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22f4ab545e034fdbb4cfa9a872941fd9"
I20260926 14:45:08.971976 7544 raft_consensus.cc:3037] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:08.971977 7681 raft_consensus.cc:3037] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:08.972841 7681 raft_consensus.cc:2454] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082247f502db4b6d88447f749061b795 in term 1.
I20260926 14:45:08.972862 7544 raft_consensus.cc:2454] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 082247f502db4b6d88447f749061b795 in term 1.
I20260926 14:45:08.973155 7343 leader_election.cc:302] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 082247f502db4b6d88447f749061b795, 22f4ab545e034fdbb4cfa9a872941fd9, a195a88278b243b3b6463201766eeb3d; no voters:
I20260926 14:45:08.973313 7825 raft_consensus.cc:2781] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:08.973526 7825 raft_consensus.cc:686] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 1 LEADER]: Becoming Leader. State: Replica: 082247f502db4b6d88447f749061b795, State: Running, Role: LEADER
I20260926 14:45:08.973953 7825 consensus_queue.cc:237] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } }
I20260926 14:45:08.974174 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:08.973948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.1:59175)
0926 14:45:08.974049 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:08.974104 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 14:45:08.976478 7274 catalog_manager.cc:5495] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 reported cstate change: term changed from 0 to 1, leader changed from <none> to 082247f502db4b6d88447f749061b795 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "082247f502db4b6d88447f749061b795" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "082247f502db4b6d88447f749061b795" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35681 } health_report { overall_health: HEALTHY } } }
I20260926 14:45:09.009220 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:09.005615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48044 (local address 127.0.98.1:35681)
0926 14:45:09.005742 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:09.005746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:09.005759 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:45:09.005931 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:09.005933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:09.005991 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:09.006083 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:45:09.006319 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:09.006859 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:09.007762 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:09.007992 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:09.008083 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:09.008339 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:45:09.008815 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:45:09.008903 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:09.009013 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 14:45:09.009066 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:45:09.012178 7681 raft_consensus.cc:1261] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d [term 1 FOLLOWER]: Refusing update from remote peer 082247f502db4b6d88447f749061b795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:09.012447 7544 raft_consensus.cc:1261] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 1 FOLLOWER]: Refusing update from remote peer 082247f502db4b6d88447f749061b795: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:09.012725 7833 consensus_queue.cc:1035] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a195a88278b243b3b6463201766eeb3d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:09.012969 7825 consensus_queue.cc:1035] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22f4ab545e034fdbb4cfa9a872941fd9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:09.014562 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:09.014247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.3:56275)
0926 14:45:09.014414 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:09.014475 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 14:45:09.015156 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:09.014859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.2:42417)
0926 14:45:09.015026 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:09.015083 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260926 14:45:09.017166 7836 mvcc.cc:205] Tried to move back new op lower bound from 7333617291308208128 to 7333617291159445504. Current Snapshot: MvccSnapshot[applied={T|T < 7333617291308208128}]
I20260926 14:45:09.017724 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:09.017485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.1:46805)
0926 14:45:09.017596 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:09.017653 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 14:45:09.018504 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:09.018273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.1:50193)
0926 14:45:09.018393 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:09.018442 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 14:45:09.019671 7838 mvcc.cc:205] Tried to move back new op lower bound from 7333617291308208128 to 7333617291159445504. Current Snapshot: MvccSnapshot[applied={T|T < 7333617291308208128}]
I20260926 14:45:09.023447 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:09.023180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.2:40521)
0926 14:45:09.023304 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:09.023361 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 14:45:09.024109 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:09.023845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38541 (local address 127.0.98.3:40031)
0926 14:45:09.023990 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:09.024040 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260926 14:45:09.026681 7273 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:09.026851 7273 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51174:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:09.030733 7273 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42fa761df91a4ca98f4f47806f82fc65: Sending DeleteTablet for 3 replicas of tablet be0edee4c8214ed4aa95cc91545e7634
I20260926 14:45:09.031407 7661 tablet_service.cc:1514] Processing DeleteTablet for tablet be0edee4c8214ed4aa95cc91545e7634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:09 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:45580
I20260926 14:45:09.031750 7387 tablet_service.cc:1514] Processing DeleteTablet for tablet be0edee4c8214ed4aa95cc91545e7634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:09 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:48042
I20260926 14:45:09.031852 7845 tablet_replica.cc:330] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d: stopping tablet replica
I20260926 14:45:09.032092 7524 tablet_service.cc:1514] Processing DeleteTablet for tablet be0edee4c8214ed4aa95cc91545e7634 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:09 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54292
I20260926 14:45:09.032366 7846 tablet_replica.cc:330] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795: stopping tablet replica
I20260926 14:45:09.032415 7847 tablet_replica.cc:330] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9: stopping tablet replica
I20260926 14:45:09.032491 7846 raft_consensus.cc:2229] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:45:09.032517 7847 raft_consensus.cc:2229] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:09.032645 7847 raft_consensus.cc:2258] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:09.032961 7846 raft_consensus.cc:2258] T be0edee4c8214ed4aa95cc91545e7634 P 082247f502db4b6d88447f749061b795 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:09.033179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 7235
I20260926 14:45:09.033799 7847 ts_tablet_manager.cc:1905] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:09.036021 7847 ts_tablet_manager.cc:1918] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:09.036079 7847 log.cc:1196] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/be0edee4c8214ed4aa95cc91545e7634
I20260926 14:45:09.036269 7847 ts_tablet_manager.cc:1939] T be0edee4c8214ed4aa95cc91545e7634 P 22f4ab545e034fdbb4cfa9a872941fd9: Deleting consensus metadata
I20260926 14:45:09.036960 7261 catalog_manager.cc:4841] TS 22f4ab545e034fdbb4cfa9a872941fd9 (127.0.98.2:43973): tablet be0edee4c8214ed4aa95cc91545e7634 (table test-table [id=d3d74c37314a4ed49c41c8160025ba12]) successfully deleted
I20260926 14:45:09.032001 7845 raft_consensus.cc:2229] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:09.039145 7845 raft_consensus.cc:2258] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:45:09.039904 7818 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:09.039584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35681 (local address 127.0.0.1:48056)
0926 14:45:09.039745 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:09.039795 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35681: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 14:45:09.039888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 7461
I20260926 14:45:09.039904 7845 ts_tablet_manager.cc:1905] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 14:45:09.040846 7259 catalog_manager.cc:4584] TS 082247f502db4b6d88447f749061b795 (127.0.98.1:35681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet be0edee4c8214ed4aa95cc91545e7634: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35681: connect: Connection reset by peer (error 104)
I20260926 14:45:09.042652 7845 ts_tablet_manager.cc:1918] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:09.042732 7845 log.cc:1196] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/be0edee4c8214ed4aa95cc91545e7634
I20260926 14:45:09.042991 7845 ts_tablet_manager.cc:1939] T be0edee4c8214ed4aa95cc91545e7634 P a195a88278b243b3b6463201766eeb3d: Deleting consensus metadata
I20260926 14:45:09.043694 7260 catalog_manager.cc:4841] TS a195a88278b243b3b6463201766eeb3d (127.0.98.3:35215): tablet be0edee4c8214ed4aa95cc91545e7634 (table test-table [id=d3d74c37314a4ed49c41c8160025ba12]) successfully deleted
I20260926 14:45:09.046984 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 7598
I20260926 14:45:09.052327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 6948
I20260926 14:45:09.057186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 7055
2026-09-26T14:45:09Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5361 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:45:09 dist-test-slave-s7k6 krb5kdc[7850](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:45:09 dist-test-slave-s7k6 krb5kdc[7850](info): set up 2 sockets
Sep 26 14:45:09 dist-test-slave-s7k6 krb5kdc[7850](info): commencing operation
krb5kdc: starting...
W20260926 14:45:10.011132 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:45:10 dist-test-slave-s7k6 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:10Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 14:45:10.126312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:10.235589 7863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:10.235868 7863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:10.235924 7863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:10.240016 7863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:10.240092 7863 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 14:45:10.240115 7863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:10.240140 7863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:10.240171 7863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:10.240202 7863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:45:10.245831 7863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42421
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:10.247004 7863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:10.248062 7863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:10.256256 7871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:10.256265 7868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:10.256264 7869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:10.256451 7863 server_base.cc:1034] running on GCE node
I20260926 14:45:10.257198 7863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:10.258293 7863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:10.259464 7863 hybrid_clock.cc:648] HybridClock initialized: now 1790433910259452 us; error 37 us; skew 500 ppm
Sep 26 14:45:10 dist-test-slave-s7k6 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:10.262437 7863 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 14:45:10.263492 7863 webserver.cc:460] Webserver started at http://127.0.98.62:46253/ using document root <none> and password file <none>
I20260926 14:45:10.264207 7863 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:10.264285 7863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:10.264535 7863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:10.267282 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "66eabb331e204caf88107e651612ec11"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "3a1fa6a58924b6868832aa86d18eca05"
server_key_iv: "52d3a44d96c31226975108b44a882704"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.267916 7863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "66eabb331e204caf88107e651612ec11"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "3a1fa6a58924b6868832aa86d18eca05"
server_key_iv: "52d3a44d96c31226975108b44a882704"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.272189 7863 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:45:10.274739 7878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:10.276026 7863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:45:10.276165 7863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "66eabb331e204caf88107e651612ec11"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "3a1fa6a58924b6868832aa86d18eca05"
server_key_iv: "52d3a44d96c31226975108b44a882704"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.276268 7863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:10.295292 7863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:10.298478 7863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:10.298660 7863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:10.306196 7863 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42421
I20260926 14:45:10.306252 7940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42421 every 8 connection(s)
I20260926 14:45:10.307147 7863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:10.309860 7941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:10.312356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 7863
I20260926 14:45:10.312456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:45:10.312707 392 external_mini_cluster.cc:1428] Setting key 10358c8fa30e9caca21880acfba4e02f
I20260926 14:45:10.315899 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11: Bootstrap starting.
I20260926 14:45:10.318511 7941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:10.319372 7941 log.cc:824] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:45:10 dist-test-slave-s7k6 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 14:45:10.321775 7941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11: No bootstrap required, opened a new log
I20260926 14:45:10.325613 7941 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eabb331e204caf88107e651612ec11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42421 } }
I20260926 14:45:10.325927 7941 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:10.325984 7941 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66eabb331e204caf88107e651612ec11, State: Initialized, Role: FOLLOWER
I20260926 14:45:10.326452 7941 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eabb331e204caf88107e651612ec11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42421 } }
I20260926 14:45:10.326582 7941 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:10.326640 7941 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:10.326732 7941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:10.326995 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:10.314100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58868 (local address 127.0.98.62:42421)
0926 14:45:10.314494 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:10.314509 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:10.314544 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:45:10.315214 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:10.315233 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:10.315592 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:10.315877 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:10.316775 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.317764 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.318922 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.319409 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.319565 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:10.322437 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:10.322442 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:10.322456 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:10.322483 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:10.324056 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:10.324551 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:10.324556 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:10.324560 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:10.324672 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:10.324933 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:10.324935 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:10.324936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:10.325347 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:10.325495 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:10.325821 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 14:45:10.326122 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":102,"threads_started":1}
I20260926 14:45:10.327667 7941 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eabb331e204caf88107e651612ec11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42421 } }
I20260926 14:45:10.327986 7941 leader_election.cc:302] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66eabb331e204caf88107e651612ec11; no voters:
I20260926 14:45:10.328281 7941 leader_election.cc:288] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:45:10.328363 7946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:10.328536 7946 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [term 1 LEADER]: Becoming Leader. State: Replica: 66eabb331e204caf88107e651612ec11, State: Running, Role: LEADER
I20260926 14:45:10.328825 7946 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eabb331e204caf88107e651612ec11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42421 } }
I20260926 14:45:10.329545 7941 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:10.330418 7948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66eabb331e204caf88107e651612ec11. Latest consensus state: current_term: 1 leader_uuid: "66eabb331e204caf88107e651612ec11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eabb331e204caf88107e651612ec11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42421 } } }
I20260926 14:45:10.330540 7948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:10.330844 7947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66eabb331e204caf88107e651612ec11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66eabb331e204caf88107e651612ec11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42421 } } }
I20260926 14:45:10.330932 7947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:10.331171 7955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:10.333827 7955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:10.339551 7955 catalog_manager.cc:1347] Generated new cluster ID: eb074e486f7548d390cf6b962d020d5a
I20260926 14:45:10.339609 7955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:10.348701 7955 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:45:10.349609 7955 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:10.359205 7955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11: Generated new TSK 0
I20260926 14:45:10.359961 7955 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 14:45:10.387827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42421
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:10.507266 7967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:10.507537 7967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:10.507607 7967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:10.511664 7967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:10.511726 7967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:10.511765 7967 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:45:10.511796 7967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:45:10.517084 7967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:10.518191 7967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:10.519238 7967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:10.528718 7975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:10.528723 7972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:10.528795 7973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:10.529201 7967 server_base.cc:1034] running on GCE node
I20260926 14:45:10.529561 7967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:10.530175 7967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:10.531325 7967 hybrid_clock.cc:648] HybridClock initialized: now 1790433910531295 us; error 42 us; skew 500 ppm
Sep 26 14:45:10 dist-test-slave-s7k6 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:10.534003 7967 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 14:45:10.534945 7967 webserver.cc:460] Webserver started at http://127.0.98.1:41795/ using document root <none> and password file <none>
I20260926 14:45:10.535477 7967 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:10.535543 7967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:10.535735 7967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:10.537715 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "1d9a124c493643b489d193521641a9e3"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "dcd145f0fb6a943c79fb72f59877f986"
server_key_iv: "9682d22ed824d6c37d19644aa0022e2f"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.538255 7967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1d9a124c493643b489d193521641a9e3"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "dcd145f0fb6a943c79fb72f59877f986"
server_key_iv: "9682d22ed824d6c37d19644aa0022e2f"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.541895 7967 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:45:10.544422 7982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:10.545699 7967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:45:10.545810 7967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "1d9a124c493643b489d193521641a9e3"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "dcd145f0fb6a943c79fb72f59877f986"
server_key_iv: "9682d22ed824d6c37d19644aa0022e2f"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.545910 7967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:10.574111 7967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:10.577288 7967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:10.577448 7967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:10.578979 7967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:10.579036 7967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:10.579077 7967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:10.579092 7967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:45:10 dist-test-slave-s7k6 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 14:45:10.589124 7967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34317
I20260926 14:45:10.589172 8097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34317 every 8 connection(s)
I20260926 14:45:10.589995 7967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:45:10.592584 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:10.579680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43587 (local address 127.0.98.62:42421)
0926 14:45:10.579832 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:10.579837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:10.580816 (+ 979us) server_negotiation.cc:407] Connection header received
0926 14:45:10.582036 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:10.582041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:10.582102 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:10.582198 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:10.583481 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.584019 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.586289 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.586532 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.586624 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:10.589210 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:10.589214 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:10.589217 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:10.589247 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:10.590723 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:10.591216 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:10.591219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:10.591220 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:10.591294 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:10.591597 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:10.591600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:10.591601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:10.591792 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:10.591879 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:10.592351 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 14:45:10.592460 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:45:10.593324 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:10.580099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42421 (local address 127.0.98.1:43587)
0926 14:45:10.580632 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:10.580692 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:10.581769 (+ 1077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:10.582411 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:10.582418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:10.582795 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:10.583330 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:10.583345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.584141 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.584143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:10.586158 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:10.586169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.586688 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.586692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:10.586890 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:10.587471 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:10.587488 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:10.589024 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:10.590860 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:10.590866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:10.590878 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:10.591140 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:10.591363 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:10.591365 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:10.591366 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:10.591483 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:10.591878 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:10.591882 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:10.592105 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 14:45:10.592250 (+ 145us) client_negotiation.cc:240] Negotiation successful
0926 14:45:10.592483 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":110,"threads_started":1}
I20260926 14:45:10.594703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 7967
I20260926 14:45:10.594704 8098 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42421
I20260926 14:45:10.594841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:45:10.594878 8098 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:10.595173 392 external_mini_cluster.cc:1428] Setting key f6fb6fdad140be1653d158dfb25dd3ac
I20260926 14:45:10.595511 8098 heartbeater.cc:511] Master 127.0.98.62:42421 requested a full tablet report, sending...
I20260926 14:45:10.597134 7895 ts_manager.cc:194] Registered new tserver with Master: 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317)
I20260926 14:45:10.598687 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43587
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 14:45:10.604897 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:10.596813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35767 (local address 127.0.98.62:42421)
0926 14:45:10.596918 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:10.596922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:10.597168 (+ 246us) server_negotiation.cc:407] Connection header received
0926 14:45:10.597301 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:10.597305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:10.597349 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:10.597393 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:10.597978 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.598731 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.599926 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.600242 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.600346 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:10.601643 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:10.601647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:10.601650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:10.601672 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:10.603148 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:10.603639 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:10.603642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:10.603643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:10.603722 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:10.604048 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:10.604052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:10.604053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:10.604235 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:10.604317 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:10.604642 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 14:45:10.604767 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:45:10.604934 8104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:10.596754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42421 (local address 127.0.98.1:35767)
0926 14:45:10.597076 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:10.597090 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:10.597190 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:10.597526 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:10.597528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:10.597737 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:10.597855 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:10.597863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.598933 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.598936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:10.599799 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:10.599811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.600369 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.600372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:10.600466 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:10.601018 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:10.601033 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:10.601484 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:10.603257 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:10.603261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:10.603263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:10.603498 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:10.603819 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:10.603823 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:10.603825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:10.603924 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:10.604338 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:10.604341 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:10.604441 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 14:45:10.604549 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 14:45:10.604723 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":96,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 14:45:10.621395 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42421
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:10.731840 8107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:10.732110 8107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:10.732172 8107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:10.736253 8107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:10.736316 8107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:10.736378 8107 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:45:10.736434 8107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:45:10.741729 8107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:10.742831 8107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:10.743850 8107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:10.753260 8115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:10.753275 8113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:10.753337 8107 server_base.cc:1034] running on GCE node
W20260926 14:45:10.753284 8112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:10.753820 8107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:10.754423 8107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:10.755574 8107 hybrid_clock.cc:648] HybridClock initialized: now 1790433910755551 us; error 39 us; skew 500 ppm
Sep 26 14:45:10 dist-test-slave-s7k6 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:10.758383 8107 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 14:45:10.759361 8107 webserver.cc:460] Webserver started at http://127.0.98.2:46477/ using document root <none> and password file <none>
I20260926 14:45:10.759927 8107 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:10.759995 8107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:10.760215 8107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:10.762313 8107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "28d1abb1abc140b295d86feb95ffd288"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "e167faf27903582fcf1d6fcbb1fdc089"
server_key_iv: "b33f191243c0e105af5a4d6ffe1f1f1d"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.762892 8107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "28d1abb1abc140b295d86feb95ffd288"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "e167faf27903582fcf1d6fcbb1fdc089"
server_key_iv: "b33f191243c0e105af5a4d6ffe1f1f1d"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.767823 8107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 14:45:10.770350 8122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:10.771695 8107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:45:10.771821 8107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "28d1abb1abc140b295d86feb95ffd288"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "e167faf27903582fcf1d6fcbb1fdc089"
server_key_iv: "b33f191243c0e105af5a4d6ffe1f1f1d"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.771926 8107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:10.784205 8107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:10.787217 8107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:10.787395 8107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:10.788882 8107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:10.788934 8107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:10.788987 8107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:10.789016 8107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:45:10 dist-test-slave-s7k6 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 14:45:10.798789 8107 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40339
I20260926 14:45:10.798871 8237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40339 every 8 connection(s)
I20260926 14:45:10.799727 8107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:45:10.801860 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:10.789645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52301 (local address 127.0.98.62:42421)
0926 14:45:10.789800 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:10.789806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:10.790665 (+ 859us) server_negotiation.cc:407] Connection header received
0926 14:45:10.791732 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:10.791736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:10.791798 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:10.791901 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:10.793141 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.793940 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.795365 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.795591 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.795690 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:10.798280 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:10.798284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:10.798287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:10.798316 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:10.799781 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:10.800319 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:10.800322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:10.800324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:10.800421 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:10.800797 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:10.800800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:10.800801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:10.801009 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:10.801098 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:10.801621 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 14:45:10.801736 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:45:10.802659 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:10.790031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42421 (local address 127.0.98.2:52301)
0926 14:45:10.790498 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:10.790550 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:10.791458 (+ 908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:10.792104 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:10.792114 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:10.792473 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:10.793000 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:10.793015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.794095 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.794099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:10.795247 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:10.795259 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.795754 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.795758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:10.795963 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:10.796536 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:10.796552 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:10.798100 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:10.799892 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:10.799898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:10.799914 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:10.800200 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:10.800526 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:10.800529 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:10.800531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:10.800708 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:10.801113 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:10.801120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:10.801368 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 14:45:10.801529 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 14:45:10.801806 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260926 14:45:10.803925 8238 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42421
I20260926 14:45:10.804065 8238 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:10.804623 8238 heartbeater.cc:511] Master 127.0.98.62:42421 requested a full tablet report, sending...
I20260926 14:45:10.805781 7893 ts_manager.cc:194] Registered new tserver with Master: 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2:40339)
I20260926 14:45:10.806511 7893 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52301
I20260926 14:45:10.807452 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 8107
I20260926 14:45:10.807571 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:45:10.807884 392 external_mini_cluster.cc:1428] Setting key cb4dd0d853297205e53745e19bd7eaa3
I20260926 14:45:10.813444 8244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:10.806122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42421 (local address 127.0.98.2:41503)
0926 14:45:10.806388 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:10.806401 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:10.806495 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:10.806711 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:10.806714 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:10.806936 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:10.807032 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:10.807038 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.807869 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.807872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:10.808521 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:10.808531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.809162 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.809165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:10.809258 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:10.809802 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:10.809813 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:10.810111 (+ 298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:10.811890 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:10.811892 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:10.811894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:10.812084 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:10.812378 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:10.812380 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:10.812381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:10.812452 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:10.812920 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:10.812922 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:10.813010 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 14:45:10.813140 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 14:45:10.813299 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":94,"threads_started":1}
I20260926 14:45:10.813499 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:10.806219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41503 (local address 127.0.98.62:42421)
0926 14:45:10.806334 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:10.806339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:10.806464 (+ 125us) server_negotiation.cc:407] Connection header received
0926 14:45:10.806568 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:10.806570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:10.806614 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:10.806692 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:10.807189 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.807752 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.808651 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:10.808942 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:10.809012 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:10.810234 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:10.810238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:10.810241 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:10.810265 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:10.811768 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:10.812207 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:10.812211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:10.812213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:10.812300 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:10.812520 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:10.812523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:10.812524 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:10.812788 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:10.812893 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:10.813219 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 14:45:10.813341 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 14:45:10.833353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42421
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:10.944903 8247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:10.945253 8247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:10.945312 8247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:10.949478 8247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:10.949554 8247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:10.949609 8247 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 14:45:10.949661 8247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:45:10.955089 8247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:10.956131 8247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:10.957114 8247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:10.967147 8252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:10.967235 8255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:10.967423 8247 server_base.cc:1034] running on GCE node
W20260926 14:45:10.967161 8253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:10.967867 8247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:10.968530 8247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:10.969702 8247 hybrid_clock.cc:648] HybridClock initialized: now 1790433910969670 us; error 44 us; skew 500 ppm
Sep 26 14:45:10 dist-test-slave-s7k6 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:10.972450 8247 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 14:45:10.973412 8247 webserver.cc:460] Webserver started at http://127.0.98.3:45827/ using document root <none> and password file <none>
I20260926 14:45:10.973976 8247 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:10.974040 8247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:10.974237 8247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:10.976413 8247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "8ac03279ec964031bb8a706d4b052b6c"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "6abbc9160848cd56dc60b7f79895a879"
server_key_iv: "906c4fec65f2dd939bb145d3806c3b57"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.976933 8247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8ac03279ec964031bb8a706d4b052b6c"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "6abbc9160848cd56dc60b7f79895a879"
server_key_iv: "906c4fec65f2dd939bb145d3806c3b57"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.980679 8247 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:45:10.983148 8262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:10.984427 8247 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:45:10.984542 8247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "8ac03279ec964031bb8a706d4b052b6c"
format_stamp: "Formatted at 2026-09-26 14:45:10 on dist-test-slave-s7k6"
server_key: "6abbc9160848cd56dc60b7f79895a879"
server_key_iv: "906c4fec65f2dd939bb145d3806c3b57"
server_key_version: "encryptionkey@0"
I20260926 14:45:10.984642 8247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:11.016952 8247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:11.020295 8247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:11.020474 8247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:11.021926 8247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:11.021977 8247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:11.022045 8247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:11.022080 8247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 14:45:11 dist-test-slave-s7k6 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 14:45:11.032442 8247 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44127
I20260926 14:45:11.032862 8377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44127 every 8 connection(s)
I20260926 14:45:11.033551 8247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:45:11.035482 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.022927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49967 (local address 127.0.98.62:42421)
0926 14:45:11.023050 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.023054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.024278 (+ 1224us) server_negotiation.cc:407] Connection header received
0926 14:45:11.025398 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.025402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.025461 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.025534 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:11.026820 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.027508 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.028676 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.028917 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.029004 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.031761 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:11.031765 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:11.031768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:11.031797 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:11.033408 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:11.033932 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:11.033936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:11.033938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:11.034031 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:11.034385 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:11.034389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:11.034390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:11.034574 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:11.034662 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.035187 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.035318 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:45:11.036283 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.023589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42421 (local address 127.0.98.3:49967)
0926 14:45:11.024118 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.024163 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.025187 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.025713 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.025721 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.026063 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:11.026655 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.026671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.027625 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.027629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.028536 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.028548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.029074 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.029078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.029250 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.029956 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:11.029972 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:11.031597 (+ 1625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:11.033519 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:11.033524 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:11.033535 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:11.033835 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:11.034135 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:11.034138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:11.034140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:11.034301 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:11.034671 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:11.034678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:11.034960 (+ 282us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.035126 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.035451 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":145,"threads_started":1}
I20260926 14:45:11.037492 8378 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42421
I20260926 14:45:11.037658 8378 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:11.038193 8378 heartbeater.cc:511] Master 127.0.98.62:42421 requested a full tablet report, sending...
I20260926 14:45:11.039283 7895 ts_manager.cc:194] Registered new tserver with Master: 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127)
I20260926 14:45:11.039911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 8247
I20260926 14:45:11.040026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:45:11.040028 7895 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49967
I20260926 14:45:11.040364 392 external_mini_cluster.cc:1428] Setting key 4091e33c2262e77cf64a9dddb2bf8253
I20260926 14:45:11.042279 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:45:11.047871 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.039519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54543 (local address 127.0.98.62:42421)
0926 14:45:11.039646 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.039650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.040885 (+ 1235us) server_negotiation.cc:407] Connection header received
0926 14:45:11.040986 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.040989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.041035 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.041118 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:11.041672 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.042377 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.043295 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.043515 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.043597 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.044675 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:11.044679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:11.044681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:11.044705 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:11.046214 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:11.046686 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:11.046690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:11.046692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:11.046813 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:11.047057 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:11.047061 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:11.047062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:11.047281 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:11.047371 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.047658 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.047754 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:45:11.047928 8384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.039526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42421 (local address 127.0.98.3:54543)
0926 14:45:11.040793 (+ 1267us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.040807 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.040905 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.041230 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.041233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.041450 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:11.041564 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.041572 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.042488 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.042492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.043191 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.043200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.043613 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.043617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.043705 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.044211 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:11.044225 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:11.044553 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:11.046351 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:11.046354 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:11.046356 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:11.046591 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:11.046896 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:11.046899 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:11.046900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:11.046978 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:11.047370 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:11.047374 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:11.047475 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.047585 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.047734 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"thread_start_us":111,"threads_started":1}
Sep 26 14:45:11 dist-test-slave-s7k6 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433911, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 14:45:11 dist-test-slave-s7k6 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:11 dist-test-slave-s7k6 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:45:11.065933 7944 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:11.060149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.0.98.62:42421)
0926 14:45:11.060304 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.060308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.060320 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:45:11.060355 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.060357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.060394 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.060463 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:11.061185 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.061697 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.062539 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.062749 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.062868 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.065815 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58870: BlockingRecv error: recv got EOF from 127.0.0.1:58870 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 14:45:11 dist-test-slave-s7k6 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433911, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 14:45:11.074848 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.067397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.0.98.62:42421)
0926 14:45:11.067511 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.067516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.067589 (+ 73us) server_negotiation.cc:407] Connection header received
0926 14:45:11.067699 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.067702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.067749 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.067813 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:11.068367 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.068859 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.069651 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.069860 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.069941 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.072057 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:11.072060 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:11.072062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:11.072084 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:11.073316 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:11.073676 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:11.073680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:11.073682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:11.073761 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:11.073991 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:11.073994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:11.073996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:11.074164 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:11.074282 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.074495 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.074599 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:45:11.077637 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:45:11.081049 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:45:11.095609 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.090290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34532 (local address 127.0.98.1:34317)
0926 14:45:11.090960 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.090973 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.090993 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:45:11.091077 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.091083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.091299 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.091471 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:11.091739 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.092696 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.094204 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.095000 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.095156 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.095312 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.095373 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.095461 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":77,"threads_started":1}
I20260926 14:45:11.095675 8401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.089928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34317 (local address 127.0.0.1:34532)
0926 14:45:11.090687 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.090700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.090815 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.091442 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.091448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.091469 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:11.091622 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.091629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.092795 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.092799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.094062 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.094073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.095104 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.095108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.095208 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.095249 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.095348 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.095467 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":56,"thread_start_us":110,"threads_started":1}
I20260926 14:45:11.095916 8404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.090590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44127 (local address 127.0.0.1:57878)
0926 14:45:11.091167 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.091179 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.091257 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.091899 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.091902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.091919 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:11.092004 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.092010 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.093329 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.093333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.094528 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.094537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.095577 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.095580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.095673 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.095695 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.095744 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.095806 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":97,"threads_started":1}
I20260926 14:45:11.096529 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.090978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57878 (local address 127.0.98.3:44127)
0926 14:45:11.091476 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.091487 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.091509 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:45:11.091579 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.091583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.091763 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.091911 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:11.092128 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.093211 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.094652 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.095434 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.096105 (+ 671us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.096257 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.096304 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.096396 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":88,"threads_started":1}
I20260926 14:45:11.096539 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.089842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40339 (local address 127.0.0.1:54016)
0926 14:45:11.090462 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.090516 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.090675 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.092068 (+ 1393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.092071 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.092088 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:11.092236 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.092242 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.093547 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.093551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.094928 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.094939 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.096165 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.096168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.096267 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.096288 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.096351 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.096415 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260926 14:45:11.096964 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.089978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54016 (local address 127.0.98.2:40339)
0926 14:45:11.090684 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.090698 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.090726 (+ 28us) server_negotiation.cc:407] Connection header received
0926 14:45:11.091642 (+ 916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.091648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.091836 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.091976 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:11.092359 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.093433 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.095045 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.095813 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.095958 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.096109 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.096749 (+ 640us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.096848 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":93,"threads_started":1}
I20260926 14:45:11.098616 8032 tablet_service.cc:1467] Processing CreateTablet for tablet e44f2183c9b744ef8996591d6510113d (DEFAULT_TABLE table=test-table [id=8b442f15de824cde8aee0818d9073a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.099157 8172 tablet_service.cc:1467] Processing CreateTablet for tablet e44f2183c9b744ef8996591d6510113d (DEFAULT_TABLE table=test-table [id=8b442f15de824cde8aee0818d9073a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.099790 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44f2183c9b744ef8996591d6510113d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.100039 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44f2183c9b744ef8996591d6510113d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.100723 8312 tablet_service.cc:1467] Processing CreateTablet for tablet e44f2183c9b744ef8996591d6510113d (DEFAULT_TABLE table=test-table [id=8b442f15de824cde8aee0818d9073a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.101722 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44f2183c9b744ef8996591d6510113d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.106402 8406 tablet_bootstrap.cc:492] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: Bootstrap starting.
I20260926 14:45:11.107915 8407 tablet_bootstrap.cc:492] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3: Bootstrap starting.
I20260926 14:45:11.108669 8406 tablet_bootstrap.cc:654] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.109472 8406 log.cc:824] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:11.109582 8408 tablet_bootstrap.cc:492] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: Bootstrap starting.
I20260926 14:45:11.110196 8407 tablet_bootstrap.cc:654] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.111027 8407 log.cc:824] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:11.111227 8406 tablet_bootstrap.cc:492] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: No bootstrap required, opened a new log
I20260926 14:45:11.111410 8406 ts_tablet_manager.cc:1397] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:45:11.111639 8408 tablet_bootstrap.cc:654] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.112417 8408 log.cc:824] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:11.112984 8407 tablet_bootstrap.cc:492] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3: No bootstrap required, opened a new log
I20260926 14:45:11.113150 8407 ts_tablet_manager.cc:1397] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:45:11.114383 8408 tablet_bootstrap.cc:492] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: No bootstrap required, opened a new log
I20260926 14:45:11.114562 8408 ts_tablet_manager.cc:1397] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:45:11.115110 8406 raft_consensus.cc:348] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.115327 8406 raft_consensus.cc:374] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.115386 8406 raft_consensus.cc:729] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d1abb1abc140b295d86feb95ffd288, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.115792 8406 consensus_queue.cc:260] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.116432 8238 heartbeater.cc:503] Master 127.0.98.62:42421 was elected leader, sending a full tablet report...
I20260926 14:45:11.116701 8406 ts_tablet_manager.cc:1428] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:45:11.117233 8407 raft_consensus.cc:348] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.117491 8407 raft_consensus.cc:374] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.117552 8407 raft_consensus.cc:729] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9a124c493643b489d193521641a9e3, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.117972 8407 consensus_queue.cc:260] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.118958 8407 ts_tablet_manager.cc:1428] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 14:45:11.119205 8098 heartbeater.cc:503] Master 127.0.98.62:42421 was elected leader, sending a full tablet report...
I20260926 14:45:11.119598 8408 raft_consensus.cc:348] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.119831 8408 raft_consensus.cc:374] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.119896 8408 raft_consensus.cc:729] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac03279ec964031bb8a706d4b052b6c, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.120324 8408 consensus_queue.cc:260] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.121201 8378 heartbeater.cc:503] Master 127.0.98.62:42421 was elected leader, sending a full tablet report...
I20260926 14:45:11.121327 8408 ts_tablet_manager.cc:1428] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 14:45:11.237069 8413 raft_consensus.cc:482] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:45:11.237275 8413 raft_consensus.cc:504] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.238488 8413 leader_election.cc:288] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2:40339), 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127)
I20260926 14:45:11.242661 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.238911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44527 (local address 127.0.98.2:40339)
0926 14:45:11.239045 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.239050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.239191 (+ 141us) server_negotiation.cc:407] Connection header received
0926 14:45:11.239385 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.239387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.239433 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.239508 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:11.239836 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.240472 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.241660 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.242096 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.242178 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.242255 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.242472 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.242540 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:45:11.242671 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.238808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40339 (local address 127.0.98.1:44527)
0926 14:45:11.239104 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.239119 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.239243 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.239565 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.239568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.239590 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:11.239717 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.239725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.240587 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.240590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.241529 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.241538 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.242255 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.242259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.242357 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.242381 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.242450 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.242519 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":111,"threads_started":1}
I20260926 14:45:11.242818 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.239447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34721 (local address 127.0.98.3:44127)
0926 14:45:11.239569 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.239574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.239728 (+ 154us) server_negotiation.cc:407] Connection header received
0926 14:45:11.239814 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.239817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.239866 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.239921 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:11.240210 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.240795 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.241805 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.242213 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.242312 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.242373 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.242594 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.242674 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:45:11.243152 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.239299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44127 (local address 127.0.98.1:34721)
0926 14:45:11.239632 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.239645 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.239738 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.239965 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.239968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.239984 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:11.240103 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.240110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.240900 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.240903 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.241689 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.241698 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.242333 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.242337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.242438 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.242461 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.242952 (+ 491us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.243021 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":104,"threads_started":1}
I20260926 14:45:11.243389 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44f2183c9b744ef8996591d6510113d" candidate_uuid: "1d9a124c493643b489d193521641a9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28d1abb1abc140b295d86feb95ffd288" is_pre_election: true
I20260926 14:45:11.243744 8192 raft_consensus.cc:2454] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d9a124c493643b489d193521641a9e3 in term 0.
I20260926 14:45:11.244033 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44f2183c9b744ef8996591d6510113d" candidate_uuid: "1d9a124c493643b489d193521641a9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac03279ec964031bb8a706d4b052b6c" is_pre_election: true
I20260926 14:45:11.244240 7985 leader_election.cc:302] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9a124c493643b489d193521641a9e3, 28d1abb1abc140b295d86feb95ffd288; no voters:
I20260926 14:45:11.244334 8332 raft_consensus.cc:2454] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d9a124c493643b489d193521641a9e3 in term 0.
I20260926 14:45:11.244489 8413 raft_consensus.cc:2781] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:45:11.244589 8413 raft_consensus.cc:482] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:45:11.244635 8413 raft_consensus.cc:3037] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.245743 8413 raft_consensus.cc:504] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.246050 8413 leader_election.cc:288] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [CANDIDATE]: Term 1 election: Requested vote from peers 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2:40339), 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127)
I20260926 14:45:11.246464 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44f2183c9b744ef8996591d6510113d" candidate_uuid: "1d9a124c493643b489d193521641a9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac03279ec964031bb8a706d4b052b6c"
I20260926 14:45:11.246491 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44f2183c9b744ef8996591d6510113d" candidate_uuid: "1d9a124c493643b489d193521641a9e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28d1abb1abc140b295d86feb95ffd288"
I20260926 14:45:11.246574 8332 raft_consensus.cc:3037] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.246592 8192 raft_consensus.cc:3037] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.247640 8192 raft_consensus.cc:2454] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d9a124c493643b489d193521641a9e3 in term 1.
I20260926 14:45:11.247645 8332 raft_consensus.cc:2454] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d9a124c493643b489d193521641a9e3 in term 1.
I20260926 14:45:11.247916 7985 leader_election.cc:302] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9a124c493643b489d193521641a9e3, 8ac03279ec964031bb8a706d4b052b6c; no voters:
I20260926 14:45:11.248092 8413 raft_consensus.cc:2781] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:11.248309 8413 raft_consensus.cc:686] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 1 LEADER]: Becoming Leader. State: Replica: 1d9a124c493643b489d193521641a9e3, State: Running, Role: LEADER
I20260926 14:45:11.248620 8413 consensus_queue.cc:237] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.251641 7895 catalog_manager.cc:5495] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d9a124c493643b489d193521641a9e3 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1d9a124c493643b489d193521641a9e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } health_report { overall_health: UNKNOWN } } }
W20260926 14:45:11.284858 8379 tablet.cc:2369] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:45:11.288970 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.285224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34540 (local address 127.0.98.1:34317)
0926 14:45:11.285373 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.285378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.285394 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:45:11.285488 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.285490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.285561 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.285651 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:45:11.285938 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.286486 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.287388 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.287646 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.287723 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.288031 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:45:11.288493 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:45:11.288583 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.288777 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.288840 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 14:45:11.294821 8332 raft_consensus.cc:1261] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Refusing update from remote peer 1d9a124c493643b489d193521641a9e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:11.294821 8192 raft_consensus.cc:1261] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Refusing update from remote peer 1d9a124c493643b489d193521641a9e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:11.295504 8417 consensus_queue.cc:1035] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:11.295699 8413 consensus_queue.cc:1035] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:45:11.301186 8239 tablet.cc:2369] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:45:11.304592 8422 mvcc.cc:205] Tried to move back new op lower bound from 7333617300656205824 to 7333617300477030400. Current Snapshot: MvccSnapshot[applied={T|T < 7333617300656205824}]
I20260926 14:45:11.306226 8420 mvcc.cc:205] Tried to move back new op lower bound from 7333617300656205824 to 7333617300477030400. Current Snapshot: MvccSnapshot[applied={T|T < 7333617300656205824 or (T in {7333617300656205824})}]
I20260926 14:45:11.306419 8421 mvcc.cc:205] Tried to move back new op lower bound from 7333617300656205824 to 7333617300477030400. Current Snapshot: MvccSnapshot[applied={T|T < 7333617300656205824}]
I20260926 14:45:11.311986 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:11.312147 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:11.314618 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11: Sending DeleteTablet for 3 replicas of tablet e44f2183c9b744ef8996591d6510113d
I20260926 14:45:11.315858 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet e44f2183c9b744ef8996591d6510113d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57878
I20260926 14:45:11.315938 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet e44f2183c9b744ef8996591d6510113d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34532
I20260926 14:45:11.315946 8172 tablet_service.cc:1514] Processing DeleteTablet for tablet e44f2183c9b744ef8996591d6510113d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54016
I20260926 14:45:11.316502 8430 tablet_replica.cc:330] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: stopping tablet replica
I20260926 14:45:11.316742 8431 tablet_replica.cc:330] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3: stopping tablet replica
I20260926 14:45:11.316951 8432 tablet_replica.cc:330] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: stopping tablet replica
I20260926 14:45:11.317318 8432 raft_consensus.cc:2229] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:11.317397 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:45:11.317824 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:45:11.318109 8431 raft_consensus.cc:2229] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:45:11.318430 8430 raft_consensus.cc:2229] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:11.318877 8432 raft_consensus.cc:2258] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:11.319695 8431 raft_consensus.cc:2258] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:11.319878 8430 raft_consensus.cc:2258] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:11.323076 8432 ts_tablet_manager.cc:1905] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:11.323076 8430 ts_tablet_manager.cc:1905] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:11.323267 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 48770606bf0f41c8ab8aa4356f04c8cc (DEFAULT_TABLE table=test-table [id=5cca2b0358814d5396ce27730160e3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.323540 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770606bf0f41c8ab8aa4356f04c8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.325803 8430 ts_tablet_manager.cc:1918] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:11.325886 8407 tablet_bootstrap.cc:492] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3: Bootstrap starting.
I20260926 14:45:11.326154 8432 ts_tablet_manager.cc:1918] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:11.326330 8432 log.cc:1196] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/e44f2183c9b744ef8996591d6510113d
I20260926 14:45:11.326689 8432 ts_tablet_manager.cc:1939] T e44f2183c9b744ef8996591d6510113d P 28d1abb1abc140b295d86feb95ffd288: Deleting consensus metadata
I20260926 14:45:11.326920 8407 tablet_bootstrap.cc:654] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.327188 8172 tablet_service.cc:1467] Processing CreateTablet for tablet 48770606bf0f41c8ab8aa4356f04c8cc (DEFAULT_TABLE table=test-table [id=5cca2b0358814d5396ce27730160e3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.327353 8431 ts_tablet_manager.cc:1905] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:11.327438 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770606bf0f41c8ab8aa4356f04c8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.328297 7881 catalog_manager.cc:4841] TS 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2:40339): tablet e44f2183c9b744ef8996591d6510113d (table test-table [id=8b442f15de824cde8aee0818d9073a8e]) successfully deleted
I20260926 14:45:11.326519 8312 tablet_service.cc:1467] Processing CreateTablet for tablet 48770606bf0f41c8ab8aa4356f04c8cc (DEFAULT_TABLE table=test-table [id=5cca2b0358814d5396ce27730160e3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.328881 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48770606bf0f41c8ab8aa4356f04c8cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.329857 8406 tablet_bootstrap.cc:492] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288: Bootstrap starting.
I20260926 14:45:11.330866 8406 tablet_bootstrap.cc:654] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.331333 8408 tablet_bootstrap.cc:492] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c: Bootstrap starting.
I20260926 14:45:11.331513 8431 ts_tablet_manager.cc:1918] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:11.331602 8431 log.cc:1196] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/wal/wals/e44f2183c9b744ef8996591d6510113d
I20260926 14:45:11.331893 8431 ts_tablet_manager.cc:1939] T e44f2183c9b744ef8996591d6510113d P 1d9a124c493643b489d193521641a9e3: Deleting consensus metadata
I20260926 14:45:11.332098 8407 tablet_bootstrap.cc:492] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3: No bootstrap required, opened a new log
I20260926 14:45:11.332170 8407 ts_tablet_manager.cc:1397] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 14:45:11.332257 8408 tablet_bootstrap.cc:654] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.332571 8407 raft_consensus.cc:348] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.332691 8407 raft_consensus.cc:374] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.332722 8407 raft_consensus.cc:729] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9a124c493643b489d193521641a9e3, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.332839 8407 consensus_queue.cc:260] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.333105 8407 ts_tablet_manager.cc:1428] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:11.325908 8430 log.cc:1196] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/e44f2183c9b744ef8996591d6510113d
I20260926 14:45:11.333693 8406 tablet_bootstrap.cc:492] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288: No bootstrap required, opened a new log
I20260926 14:45:11.333784 8406 ts_tablet_manager.cc:1397] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 14:45:11.333813 8430 ts_tablet_manager.cc:1939] T e44f2183c9b744ef8996591d6510113d P 8ac03279ec964031bb8a706d4b052b6c: Deleting consensus metadata
I20260926 14:45:11.333904 7882 catalog_manager.cc:4841] TS 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317): tablet e44f2183c9b744ef8996591d6510113d (table test-table [id=8b442f15de824cde8aee0818d9073a8e]) successfully deleted
I20260926 14:45:11.334270 8406 raft_consensus.cc:348] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.334385 8406 raft_consensus.cc:374] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.334422 8406 raft_consensus.cc:729] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d1abb1abc140b295d86feb95ffd288, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.334542 8406 consensus_queue.cc:260] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.334813 8406 ts_tablet_manager.cc:1428] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:11.335373 7881 catalog_manager.cc:4841] TS 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127): tablet e44f2183c9b744ef8996591d6510113d (table test-table [id=8b442f15de824cde8aee0818d9073a8e]) successfully deleted
I20260926 14:45:11.336205 8408 tablet_bootstrap.cc:492] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c: No bootstrap required, opened a new log
I20260926 14:45:11.336283 8408 ts_tablet_manager.cc:1397] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 14:45:11.336699 8408 raft_consensus.cc:348] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.336838 8408 raft_consensus.cc:374] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.336876 8408 raft_consensus.cc:729] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac03279ec964031bb8a706d4b052b6c, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.336992 8408 consensus_queue.cc:260] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.337226 8408 ts_tablet_manager.cc:1428] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260926 14:45:11.341230 8099 tablet.cc:2369] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:45:11.478634 8414 raft_consensus.cc:482] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:45:11.478823 8414 raft_consensus.cc:504] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.480018 8414 leader_election.cc:288] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317), 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2:40339)
I20260926 14:45:11.484881 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.480419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37051 (local address 127.0.98.1:34317)
0926 14:45:11.480545 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.480550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.480625 (+ 75us) server_negotiation.cc:407] Connection header received
0926 14:45:11.480795 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.480798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.480844 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.480915 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:11.481230 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.482012 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.483745 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.484199 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.484313 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.484382 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.484675 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.484762 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:45:11.484880 8271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.480307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34317 (local address 127.0.98.3:37051)
0926 14:45:11.480542 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.480558 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.480686 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.480943 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.480945 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.480976 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:11.481109 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.481118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.482158 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.482162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.483579 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.483591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.484442 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.484446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.484554 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.484572 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.484639 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.484719 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":49}
I20260926 14:45:11.484907 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.480414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41231 (local address 127.0.98.2:40339)
0926 14:45:11.480541 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.480545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.480803 (+ 258us) server_negotiation.cc:407] Connection header received
0926 14:45:11.480919 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.480922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.480987 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.481050 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:11.481324 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.482091 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.483734 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.484222 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.484329 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.484385 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.484715 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.484792 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:45:11.484925 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.480307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40339 (local address 127.0.98.3:41231)
0926 14:45:11.480704 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.480719 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.480830 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.481069 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.481071 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.481085 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:11.481226 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.481232 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.482190 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.482193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.483586 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.483592 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.484454 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.484458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.484598 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.484619 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.484685 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.484767 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":129,"thread_start_us":99,"threads_started":1}
I20260926 14:45:11.485540 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770606bf0f41c8ab8aa4356f04c8cc" candidate_uuid: "8ac03279ec964031bb8a706d4b052b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9a124c493643b489d193521641a9e3" is_pre_election: true
I20260926 14:45:11.485778 8052 raft_consensus.cc:2454] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac03279ec964031bb8a706d4b052b6c in term 0.
I20260926 14:45:11.486052 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770606bf0f41c8ab8aa4356f04c8cc" candidate_uuid: "8ac03279ec964031bb8a706d4b052b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28d1abb1abc140b295d86feb95ffd288" is_pre_election: true
I20260926 14:45:11.486195 8192 raft_consensus.cc:2454] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac03279ec964031bb8a706d4b052b6c in term 0.
I20260926 14:45:11.486217 8266 leader_election.cc:302] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9a124c493643b489d193521641a9e3, 8ac03279ec964031bb8a706d4b052b6c; no voters:
I20260926 14:45:11.486496 8414 raft_consensus.cc:2781] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:45:11.486554 8414 raft_consensus.cc:482] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:45:11.486581 8414 raft_consensus.cc:3037] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.487619 8414 raft_consensus.cc:504] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.487973 8414 leader_election.cc:288] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [CANDIDATE]: Term 1 election: Requested vote from peers 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317), 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2:40339)
I20260926 14:45:11.488322 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770606bf0f41c8ab8aa4356f04c8cc" candidate_uuid: "8ac03279ec964031bb8a706d4b052b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9a124c493643b489d193521641a9e3"
I20260926 14:45:11.488345 8192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48770606bf0f41c8ab8aa4356f04c8cc" candidate_uuid: "8ac03279ec964031bb8a706d4b052b6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28d1abb1abc140b295d86feb95ffd288"
I20260926 14:45:11.488451 8052 raft_consensus.cc:3037] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.488451 8192 raft_consensus.cc:3037] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.489645 8192 raft_consensus.cc:2454] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac03279ec964031bb8a706d4b052b6c in term 1.
I20260926 14:45:11.489646 8052 raft_consensus.cc:2454] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ac03279ec964031bb8a706d4b052b6c in term 1.
I20260926 14:45:11.489966 8265 leader_election.cc:302] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28d1abb1abc140b295d86feb95ffd288, 8ac03279ec964031bb8a706d4b052b6c; no voters:
I20260926 14:45:11.490134 8414 raft_consensus.cc:2781] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:11.490325 8414 raft_consensus.cc:686] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 1 LEADER]: Becoming Leader. State: Replica: 8ac03279ec964031bb8a706d4b052b6c, State: Running, Role: LEADER
I20260926 14:45:11.490553 8414 consensus_queue.cc:237] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.492964 7894 catalog_manager.cc:5495] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8ac03279ec964031bb8a706d4b052b6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } health_report { overall_health: HEALTHY } } }
Sep 26 14:45:11 dist-test-slave-s7k6 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433910, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 14:45:11.552716 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.544351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42421 (local address 127.0.0.1:58892)
0926 14:45:11.544608 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.544620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.544733 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.544946 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.544948 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.545358 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:11.545455 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.545461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.546258 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.546261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.546852 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.546861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.547353 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.547356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.547423 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.548001 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:11.548013 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:11.549421 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:11.551089 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:11.551098 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:11.551103 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:11.551355 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:11.551674 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:11.551677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:11.551679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:11.551781 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:11.552207 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:11.552213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:11.552325 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.552427 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.552546 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":99,"threads_started":1}
I20260926 14:45:11.552740 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.544409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.0.98.62:42421)
0926 14:45:11.544558 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.544563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.544699 (+ 136us) server_negotiation.cc:407] Connection header received
0926 14:45:11.544821 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.544823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.544863 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.544920 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:11.545566 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.546149 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.546966 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.547176 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.547299 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.549550 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:11.549553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:11.549555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:11.549577 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:11.550972 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:11.551447 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:11.551450 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:11.551451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:11.551532 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:11.551891 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:11.551893 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:11.551894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:11.552068 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:11.552171 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.552514 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.552612 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:45:11.556543 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58892:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 14:45:11.557194 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:45:11.561794 8172 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7153a408c64bf3aca8e036825bc9b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=861fe0f63d13479d9f703ab35e194a9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:45:11.561762 8032 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7153a408c64bf3aca8e036825bc9b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=861fe0f63d13479d9f703ab35e194a9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:45:11.561847 8312 tablet_service.cc:1467] Processing CreateTablet for tablet 7b7153a408c64bf3aca8e036825bc9b2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=861fe0f63d13479d9f703ab35e194a9e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 14:45:11.562179 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7153a408c64bf3aca8e036825bc9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.562187 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7153a408c64bf3aca8e036825bc9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.562191 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b7153a408c64bf3aca8e036825bc9b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.564872 8408 tablet_bootstrap.cc:492] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c: Bootstrap starting.
I20260926 14:45:11.564872 8407 tablet_bootstrap.cc:492] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3: Bootstrap starting.
I20260926 14:45:11.564921 8406 tablet_bootstrap.cc:492] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: Bootstrap starting.
I20260926 14:45:11.565680 8407 tablet_bootstrap.cc:654] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.565696 8406 tablet_bootstrap.cc:654] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.565697 8408 tablet_bootstrap.cc:654] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.566886 8408 tablet_bootstrap.cc:492] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c: No bootstrap required, opened a new log
I20260926 14:45:11.566892 8407 tablet_bootstrap.cc:492] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3: No bootstrap required, opened a new log
I20260926 14:45:11.566963 8407 ts_tablet_manager.cc:1397] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:45:11.566963 8408 ts_tablet_manager.cc:1397] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:45:11.566970 8406 tablet_bootstrap.cc:492] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: No bootstrap required, opened a new log
I20260926 14:45:11.567036 8406 ts_tablet_manager.cc:1397] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:45:11.567391 8406 raft_consensus.cc:348] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.567480 8406 raft_consensus.cc:374] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.567413 8407 raft_consensus.cc:348] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.567502 8406 raft_consensus.cc:729] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d1abb1abc140b295d86feb95ffd288, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.567523 8407 raft_consensus.cc:374] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.567555 8407 raft_consensus.cc:729] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9a124c493643b489d193521641a9e3, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.567598 8406 consensus_queue.cc:260] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.567647 8408 raft_consensus.cc:348] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.567672 8407 consensus_queue.cc:260] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.567739 8408 raft_consensus.cc:374] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.567767 8408 raft_consensus.cc:729] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac03279ec964031bb8a706d4b052b6c, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.567847 8406 ts_tablet_manager.cc:1428] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:11.567824 8412 tablet_replica.cc:439] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } }
I20260926 14:45:11.567873 8408 consensus_queue.cc:260] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.567922 8407 ts_tablet_manager.cc:1428] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:45:11.567945 8412 tablet_replica.cc:442] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:45:11.568105 8408 ts_tablet_manager.cc:1428] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:45:11.568091 8426 tablet_replica.cc:439] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } }
I20260926 14:45:11.568205 8426 tablet_replica.cc:442] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:45:11.568274 8414 tablet_replica.cc:439] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } }
I20260926 14:45:11.568370 8414 tablet_replica.cc:442] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:45:11.602059 8412 raft_consensus.cc:482] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:45:11.602219 8412 raft_consensus.cc:504] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.603319 8412 leader_election.cc:288] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317), 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127)
I20260926 14:45:11.607561 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.603679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54901 (local address 127.0.98.3:44127)
0926 14:45:11.603854 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.603858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.603957 (+ 99us) server_negotiation.cc:407] Connection header received
0926 14:45:11.604140 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.604143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.604191 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.604236 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:11.604579 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.605299 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.606517 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.607052 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.607145 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.607212 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.607372 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.607445 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260926 14:45:11.607594 8403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.603683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48945 (local address 127.0.98.1:34317)
0926 14:45:11.603854 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.603858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.604296 (+ 438us) server_negotiation.cc:407] Connection header received
0926 14:45:11.604411 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.604413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.604458 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.604517 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:11.604772 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.605327 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.606501 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.607043 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.607107 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.607158 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.607417 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.607490 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 14:45:11.607606 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.603542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44127 (local address 127.0.98.2:54901)
0926 14:45:11.603865 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.603879 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.603982 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.604364 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.604367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.604383 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:11.604476 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.604481 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.605396 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.605400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.606422 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.606431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.607162 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.607165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.607264 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.607287 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.607350 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.607417 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":51,"thread_start_us":83,"threads_started":1}
I20260926 14:45:11.607612 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.603541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34317 (local address 127.0.98.2:48945)
0926 14:45:11.604204 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.604218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.604319 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.604531 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.604534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.604548 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:11.604676 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.604683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.605433 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.605436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.606377 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.606386 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.607239 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.607242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.607315 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.607330 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.607387 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.607453 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"mutex_wait_us":363,"thread_start_us":111,"threads_started":1}
I20260926 14:45:11.608134 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7153a408c64bf3aca8e036825bc9b2" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac03279ec964031bb8a706d4b052b6c" is_pre_election: true
I20260926 14:45:11.608129 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7153a408c64bf3aca8e036825bc9b2" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9a124c493643b489d193521641a9e3" is_pre_election: true
I20260926 14:45:11.608268 8052 raft_consensus.cc:2454] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 0.
I20260926 14:45:11.608276 8332 raft_consensus.cc:2454] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 0.
I20260926 14:45:11.608603 8125 leader_election.cc:302] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28d1abb1abc140b295d86feb95ffd288, 8ac03279ec964031bb8a706d4b052b6c; no voters:
I20260926 14:45:11.608867 8412 raft_consensus.cc:2781] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:45:11.608923 8412 raft_consensus.cc:482] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:45:11.608968 8412 raft_consensus.cc:3037] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.609913 8412 raft_consensus.cc:504] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.610224 8412 leader_election.cc:288] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 election: Requested vote from peers 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317), 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127)
I20260926 14:45:11.610574 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7153a408c64bf3aca8e036825bc9b2" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac03279ec964031bb8a706d4b052b6c"
I20260926 14:45:11.610683 8332 raft_consensus.cc:3037] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.610881 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b7153a408c64bf3aca8e036825bc9b2" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9a124c493643b489d193521641a9e3"
I20260926 14:45:11.610993 8052 raft_consensus.cc:3037] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.611636 8332 raft_consensus.cc:2454] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 1.
I20260926 14:45:11.611899 8052 raft_consensus.cc:2454] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 1.
I20260926 14:45:11.612178 8126 leader_election.cc:302] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d9a124c493643b489d193521641a9e3, 28d1abb1abc140b295d86feb95ffd288, 8ac03279ec964031bb8a706d4b052b6c; no voters:
I20260926 14:45:11.612386 8412 raft_consensus.cc:2781] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:11.612569 8412 raft_consensus.cc:686] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [term 1 LEADER]: Becoming Leader. State: Replica: 28d1abb1abc140b295d86feb95ffd288, State: Running, Role: LEADER
I20260926 14:45:11.612828 8412 consensus_queue.cc:237] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.613929 8450 tablet_replica.cc:439] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: TxnStatusTablet state changed. Reason: New leader 28d1abb1abc140b295d86feb95ffd288. Latest consensus state: current_term: 1 leader_uuid: "28d1abb1abc140b295d86feb95ffd288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } }
I20260926 14:45:11.614035 8450 tablet_replica.cc:442] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:45:11.614311 8452 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 14:45:11.614466 8452 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 14:45:11.615154 7893 catalog_manager.cc:5495] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "28d1abb1abc140b295d86feb95ffd288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } health_report { overall_health: UNKNOWN } } }
I20260926 14:45:11.639736 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.635586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54030 (local address 127.0.98.2:40339)
0926 14:45:11.635705 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.635708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.635801 (+ 93us) server_negotiation.cc:407] Connection header received
0926 14:45:11.636001 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.636004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.636055 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.636154 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:45:11.636389 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.636930 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.638190 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.638425 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.638526 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.638856 (+ 330us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:45:11.639349 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:45:11.639437 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.639539 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.639594 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:45:11.639778 8444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.635491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40339 (local address 127.0.0.1:54030)
0926 14:45:11.635741 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.635756 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.635870 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.636165 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.636167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.636177 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:45:11.636301 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.636307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.637042 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.637046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.638014 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.638025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.638527 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.638530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.638621 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.638675 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:45:11.639443 (+ 768us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:45:11.639456 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.639525 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.639625 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260926 14:45:11.642951 8332 raft_consensus.cc:1261] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Refusing update from remote peer 28d1abb1abc140b295d86feb95ffd288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:11.643096 8052 raft_consensus.cc:1261] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Refusing update from remote peer 28d1abb1abc140b295d86feb95ffd288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:11.643458 8450 consensus_queue.cc:1035] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:11.643627 8412 consensus_queue.cc:1035] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:11.645042 8414 tablet_replica.cc:439] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c: TxnStatusTablet state changed. Reason: New leader 28d1abb1abc140b295d86feb95ffd288. Latest consensus state: current_term: 1 leader_uuid: "28d1abb1abc140b295d86feb95ffd288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } }
I20260926 14:45:11.645156 8414 tablet_replica.cc:442] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:45:11.645407 8426 tablet_replica.cc:439] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3: TxnStatusTablet state changed. Reason: New leader 28d1abb1abc140b295d86feb95ffd288. Latest consensus state: current_term: 1 leader_uuid: "28d1abb1abc140b295d86feb95ffd288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } }
I20260926 14:45:11.645524 8426 tablet_replica.cc:442] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:45:11.646966 8412 tablet_replica.cc:439] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28d1abb1abc140b295d86feb95ffd288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } }
I20260926 14:45:11.647099 8412 tablet_replica.cc:442] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:45:11.647843 8414 tablet_replica.cc:439] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28d1abb1abc140b295d86feb95ffd288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } }
I20260926 14:45:11.647953 8414 tablet_replica.cc:442] T 7b7153a408c64bf3aca8e036825bc9b2 P 8ac03279ec964031bb8a706d4b052b6c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:45:11.648730 8450 tablet_replica.cc:439] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "28d1abb1abc140b295d86feb95ffd288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } }
I20260926 14:45:11.648836 8450 tablet_replica.cc:442] T 7b7153a408c64bf3aca8e036825bc9b2 P 28d1abb1abc140b295d86feb95ffd288: This TxnStatusTablet replica's current role is: LEADER
I20260926 14:45:11.650393 8426 tablet_replica.cc:439] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "28d1abb1abc140b295d86feb95ffd288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } } }
I20260926 14:45:11.650504 8426 tablet_replica.cc:442] T 7b7153a408c64bf3aca8e036825bc9b2 P 1d9a124c493643b489d193521641a9e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 14:45:11.657476 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.653838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.0.98.3:44127)
0926 14:45:11.653948 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.653951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.653963 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:45:11.654024 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.654026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.654071 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.654159 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:45:11.654451 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.655087 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.656193 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.656411 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.656482 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.656733 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:45:11.657139 (+ 406us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:45:11.657220 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.657306 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.657350 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:45:11.667609 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.664173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41211 (local address 127.0.98.2:40339)
0926 14:45:11.664300 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.664304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.664397 (+ 93us) server_negotiation.cc:407] Connection header received
0926 14:45:11.664498 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.664501 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.664557 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.664633 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:45:11.664844 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.665613 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.666508 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.666863 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.666956 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.667205 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:45:11.667291 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:45:11.667357 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.667427 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.667477 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:45:11.667623 8460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.664053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40339 (local address 127.0.98.3:41211)
0926 14:45:11.664323 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.664336 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.664419 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.664634 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.664636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.664641 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:45:11.664759 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.664763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.665702 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.665705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.666380 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.666409 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.666958 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.666961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.667033 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.667075 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:45:11.667361 (+ 286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:45:11.667366 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.667414 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.667477 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":89,"threads_started":1}
I20260926 14:45:11.674633 8052 raft_consensus.cc:1261] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Refusing update from remote peer 8ac03279ec964031bb8a706d4b052b6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:11.674705 8192 raft_consensus.cc:1261] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Refusing update from remote peer 8ac03279ec964031bb8a706d4b052b6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:11.675179 8437 consensus_queue.cc:1035] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:11.675381 8414 consensus_queue.cc:1035] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:11.699349 8405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.695724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60725 (local address 127.0.98.3:44127)
0926 14:45:11.695843 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.695848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.696108 (+ 260us) server_negotiation.cc:407] Connection header received
0926 14:45:11.696235 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.696237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.696279 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.696335 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:45:11.696598 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.697111 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.698323 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.698596 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.698672 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.698989 (+ 317us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:45:11.699052 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:45:11.699098 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.699193 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.699235 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:45:11.699430 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.695628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44127 (local address 127.0.98.2:60725)
0926 14:45:11.696023 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:11.696037 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:11.696151 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:11.696362 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:11.696365 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:11.696373 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:45:11.696490 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.696496 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.697207 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.697217 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.698206 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:11.698223 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.698691 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.698695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:11.698817 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.698870 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:45:11.699117 (+ 247us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:45:11.699122 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:45:11.699184 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 14:45:11.699270 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":131,"threads_started":1}
I20260926 14:45:11.723434 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:11.723623 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:11.725960 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11: Sending DeleteTablet for 3 replicas of tablet 48770606bf0f41c8ab8aa4356f04c8cc
I20260926 14:45:11.726599 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet 48770606bf0f41c8ab8aa4356f04c8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34532
I20260926 14:45:11.726815 8431 tablet_replica.cc:330] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3: stopping tablet replica
I20260926 14:45:11.726933 8431 raft_consensus.cc:2229] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:11.726975 8172 tablet_service.cc:1514] Processing DeleteTablet for tablet 48770606bf0f41c8ab8aa4356f04c8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54016
I20260926 14:45:11.727051 8431 raft_consensus.cc:2258] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:11.727150 8432 tablet_replica.cc:330] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288: stopping tablet replica
I20260926 14:45:11.727267 8432 raft_consensus.cc:2229] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:11.727383 8432 raft_consensus.cc:2258] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:11.727695 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet 48770606bf0f41c8ab8aa4356f04c8cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57878
I20260926 14:45:11.727856 8430 tablet_replica.cc:330] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c: stopping tablet replica
I20260926 14:45:11.727962 8430 raft_consensus.cc:2229] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:45:11.728183 8432 ts_tablet_manager.cc:1905] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:11.728262 8430 raft_consensus.cc:2258] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:11.728965 8430 ts_tablet_manager.cc:1905] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:11.731066 8432 ts_tablet_manager.cc:1918] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:45:11.731132 8432 log.cc:1196] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/48770606bf0f41c8ab8aa4356f04c8cc
I20260926 14:45:11.731359 8432 ts_tablet_manager.cc:1939] T 48770606bf0f41c8ab8aa4356f04c8cc P 28d1abb1abc140b295d86feb95ffd288: Deleting consensus metadata
I20260926 14:45:11.731602 8430 ts_tablet_manager.cc:1918] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:45:11.731671 8430 log.cc:1196] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/48770606bf0f41c8ab8aa4356f04c8cc
I20260926 14:45:11.731881 8430 ts_tablet_manager.cc:1939] T 48770606bf0f41c8ab8aa4356f04c8cc P 8ac03279ec964031bb8a706d4b052b6c: Deleting consensus metadata
I20260926 14:45:11.731892 8431 ts_tablet_manager.cc:1905] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:11.732026 7881 catalog_manager.cc:4841] TS 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2:40339): tablet 48770606bf0f41c8ab8aa4356f04c8cc (table test-table [id=5cca2b0358814d5396ce27730160e3e5]) successfully deleted
I20260926 14:45:11.732517 7881 catalog_manager.cc:4841] TS 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127): tablet 48770606bf0f41c8ab8aa4356f04c8cc (table test-table [id=5cca2b0358814d5396ce27730160e3e5]) successfully deleted
I20260926 14:45:11.734509 8431 ts_tablet_manager.cc:1918] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:45:11.734570 8431 log.cc:1196] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/wal/wals/48770606bf0f41c8ab8aa4356f04c8cc
I20260926 14:45:11.734993 8431 ts_tablet_manager.cc:1939] T 48770606bf0f41c8ab8aa4356f04c8cc P 1d9a124c493643b489d193521641a9e3: Deleting consensus metadata
I20260926 14:45:11.735924 7882 catalog_manager.cc:4841] TS 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317): tablet 48770606bf0f41c8ab8aa4356f04c8cc (table test-table [id=5cca2b0358814d5396ce27730160e3e5]) successfully deleted
Sep 26 14:45:11 dist-test-slave-s7k6 krb5kdc[7850](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 14:45:11 dist-test-slave-s7k6 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:11 dist-test-slave-s7k6 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:45:11.761569 7944 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:11.753511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.0.98.62:42421)
0926 14:45:11.753663 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.753668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.754006 (+ 338us) server_negotiation.cc:407] Connection header received
0926 14:45:11.754174 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.754178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.754233 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.754322 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:11.755233 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.756057 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.757343 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.757688 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.757786 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.761433 (+ 3647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58898: BlockingRecv error: recv got EOF from 127.0.0.1:58898 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 14:45:11 dist-test-slave-s7k6 krb5kdc[7850](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433911, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 14:45:11.774391 7944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.763599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58906 (local address 127.0.98.62:42421)
0926 14:45:11.763716 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.763719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.763933 (+ 214us) server_negotiation.cc:407] Connection header received
0926 14:45:11.764044 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.764047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.764098 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.764176 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:11.764886 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.765691 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.766847 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.767206 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.767313 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.770197 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:11.770201 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:11.770204 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:11.770263 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:11.772507 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:11.772998 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:11.773002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:11.773004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:11.773092 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:11.773397 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:11.773400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:11.773402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:11.773613 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:11.773707 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.774032 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.774159 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:45:11.777849 7895 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:45:11.778313 7895 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:45:11.783144 8032 tablet_service.cc:1467] Processing CreateTablet for tablet f5db2c4383f0408b960b375dcd3619e8 (DEFAULT_TABLE table=test-table [id=359ef82508de4f948cc02c5b331b4684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.783269 8172 tablet_service.cc:1467] Processing CreateTablet for tablet f5db2c4383f0408b960b375dcd3619e8 (DEFAULT_TABLE table=test-table [id=359ef82508de4f948cc02c5b331b4684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.783455 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5db2c4383f0408b960b375dcd3619e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.783535 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5db2c4383f0408b960b375dcd3619e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.785851 8406 tablet_bootstrap.cc:492] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288: Bootstrap starting.
I20260926 14:45:11.785892 8407 tablet_bootstrap.cc:492] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3: Bootstrap starting.
I20260926 14:45:11.786688 8407 tablet_bootstrap.cc:654] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.786688 8406 tablet_bootstrap.cc:654] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.787634 8312 tablet_service.cc:1467] Processing CreateTablet for tablet f5db2c4383f0408b960b375dcd3619e8 (DEFAULT_TABLE table=test-table [id=359ef82508de4f948cc02c5b331b4684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.787900 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5db2c4383f0408b960b375dcd3619e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.787974 8407 tablet_bootstrap.cc:492] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3: No bootstrap required, opened a new log
I20260926 14:45:11.788043 8407 ts_tablet_manager.cc:1397] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:45:11.788416 8407 raft_consensus.cc:348] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.788514 8407 raft_consensus.cc:374] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.788537 8407 raft_consensus.cc:729] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9a124c493643b489d193521641a9e3, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.788643 8407 consensus_queue.cc:260] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.788866 8407 ts_tablet_manager.cc:1428] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:11.790212 8408 tablet_bootstrap.cc:492] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c: Bootstrap starting.
I20260926 14:45:11.790911 8406 tablet_bootstrap.cc:492] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288: No bootstrap required, opened a new log
I20260926 14:45:11.790977 8406 ts_tablet_manager.cc:1397] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260926 14:45:11.791164 8408 tablet_bootstrap.cc:654] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.791289 8406 raft_consensus.cc:348] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.791375 8406 raft_consensus.cc:374] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.791394 8406 raft_consensus.cc:729] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d1abb1abc140b295d86feb95ffd288, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.791498 8406 consensus_queue.cc:260] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.791751 8406 ts_tablet_manager.cc:1428] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:45:11.792984 8408 tablet_bootstrap.cc:492] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c: No bootstrap required, opened a new log
I20260926 14:45:11.793069 8408 ts_tablet_manager.cc:1397] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 14:45:11.793488 8408 raft_consensus.cc:348] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.793615 8408 raft_consensus.cc:374] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.793644 8408 raft_consensus.cc:729] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac03279ec964031bb8a706d4b052b6c, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.793763 8408 consensus_queue.cc:260] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.794036 8408 ts_tablet_manager.cc:1428] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:45:11.795902 8412 raft_consensus.cc:482] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:45:11.795990 8412 raft_consensus.cc:504] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.796340 8412 leader_election.cc:288] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317), 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127)
I20260926 14:45:11.796648 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5db2c4383f0408b960b375dcd3619e8" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9a124c493643b489d193521641a9e3" is_pre_election: true
I20260926 14:45:11.796694 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5db2c4383f0408b960b375dcd3619e8" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac03279ec964031bb8a706d4b052b6c" is_pre_election: true
I20260926 14:45:11.796764 8052 raft_consensus.cc:2454] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 0.
I20260926 14:45:11.796811 8332 raft_consensus.cc:2454] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 0.
I20260926 14:45:11.797041 8126 leader_election.cc:302] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9a124c493643b489d193521641a9e3, 28d1abb1abc140b295d86feb95ffd288; no voters:
I20260926 14:45:11.797250 8412 raft_consensus.cc:2781] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:45:11.797305 8412 raft_consensus.cc:482] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:45:11.797340 8412 raft_consensus.cc:3037] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.798244 8412 raft_consensus.cc:504] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.798581 8412 leader_election.cc:288] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 election: Requested vote from peers 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317), 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127)
I20260926 14:45:11.798897 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5db2c4383f0408b960b375dcd3619e8" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9a124c493643b489d193521641a9e3"
I20260926 14:45:11.798995 8052 raft_consensus.cc:3037] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.799919 8052 raft_consensus.cc:2454] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 1.
I20260926 14:45:11.800423 8126 leader_election.cc:302] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9a124c493643b489d193521641a9e3, 28d1abb1abc140b295d86feb95ffd288; no voters:
I20260926 14:45:11.800596 8412 raft_consensus.cc:2781] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:11.800671 8412 raft_consensus.cc:686] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 1 LEADER]: Becoming Leader. State: Replica: 28d1abb1abc140b295d86feb95ffd288, State: Running, Role: LEADER
I20260926 14:45:11.800798 8412 consensus_queue.cc:237] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.802448 7894 catalog_manager.cc:5495] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "28d1abb1abc140b295d86feb95ffd288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } health_report { overall_health: UNKNOWN } } }
I20260926 14:45:11.803231 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5db2c4383f0408b960b375dcd3619e8" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac03279ec964031bb8a706d4b052b6c"
I20260926 14:45:11.803362 8332 raft_consensus.cc:3037] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.804536 8332 raft_consensus.cc:2454] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 1.
I20260926 14:45:11.815057 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:11.812001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54044 (local address 127.0.98.2:40339)
0926 14:45:11.812336 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:11.812340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:11.812349 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:45:11.812384 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:11.812386 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:11.812422 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:11.812482 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:45:11.812726 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.813234 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.814067 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:11.814292 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:11.814376 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:11.814621 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:45:11.814690 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:45:11.814747 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:11.814877 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 14:45:11.814929 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260926 14:45:11.817308 8332 raft_consensus.cc:1261] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Refusing update from remote peer 28d1abb1abc140b295d86feb95ffd288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:11.817308 8052 raft_consensus.cc:1261] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Refusing update from remote peer 28d1abb1abc140b295d86feb95ffd288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:11.817716 8457 consensus_queue.cc:1035] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:11.818006 8457 consensus_queue.cc:1035] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:11.829358 7894 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:11.829560 7894 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:11.831815 7894 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11: Sending DeleteTablet for 3 replicas of tablet f5db2c4383f0408b960b375dcd3619e8
I20260926 14:45:11.832293 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet f5db2c4383f0408b960b375dcd3619e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34532
I20260926 14:45:11.832463 8431 tablet_replica.cc:330] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3: stopping tablet replica
I20260926 14:45:11.832572 8431 raft_consensus.cc:2229] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:11.832692 8431 raft_consensus.cc:2258] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:11.833315 8172 tablet_service.cc:1514] Processing DeleteTablet for tablet f5db2c4383f0408b960b375dcd3619e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54016
I20260926 14:45:11.833468 8432 tablet_replica.cc:330] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288: stopping tablet replica
I20260926 14:45:11.833451 8431 ts_tablet_manager.cc:1905] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:11.833566 8432 raft_consensus.cc:2229] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:45:11.833828 8432 raft_consensus.cc:2258] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:11.835165 8432 ts_tablet_manager.cc:1905] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:11.835985 8431 ts_tablet_manager.cc:1918] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:11.836052 8431 log.cc:1196] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-0/wal/wals/f5db2c4383f0408b960b375dcd3619e8
I20260926 14:45:11.836290 8431 ts_tablet_manager.cc:1939] T f5db2c4383f0408b960b375dcd3619e8 P 1d9a124c493643b489d193521641a9e3: Deleting consensus metadata
I20260926 14:45:11.836903 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet f5db2c4383f0408b960b375dcd3619e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:11 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57878
I20260926 14:45:11.837062 8430 tablet_replica.cc:330] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c: stopping tablet replica
I20260926 14:45:11.837152 8430 raft_consensus.cc:2229] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:11.837284 8430 raft_consensus.cc:2258] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:11.837548 7882 catalog_manager.cc:4841] TS 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317): tablet f5db2c4383f0408b960b375dcd3619e8 (table test-table [id=359ef82508de4f948cc02c5b331b4684]) successfully deleted
I20260926 14:45:11.837628 8432 ts_tablet_manager.cc:1918] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:11.837694 8432 log.cc:1196] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/f5db2c4383f0408b960b375dcd3619e8
I20260926 14:45:11.837930 8432 ts_tablet_manager.cc:1939] T f5db2c4383f0408b960b375dcd3619e8 P 28d1abb1abc140b295d86feb95ffd288: Deleting consensus metadata
I20260926 14:45:11.838577 7881 catalog_manager.cc:4841] TS 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2:40339): tablet f5db2c4383f0408b960b375dcd3619e8 (table test-table [id=359ef82508de4f948cc02c5b331b4684]) successfully deleted
I20260926 14:45:11.838493 7894 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:45:11.839080 7894 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:45:11.839480 8430 ts_tablet_manager.cc:1905] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:11.841831 8430 ts_tablet_manager.cc:1918] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:11.841917 8430 log.cc:1196] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/f5db2c4383f0408b960b375dcd3619e8
I20260926 14:45:11.842563 8430 ts_tablet_manager.cc:1939] T f5db2c4383f0408b960b375dcd3619e8 P 8ac03279ec964031bb8a706d4b052b6c: Deleting consensus metadata
I20260926 14:45:11.843940 7881 catalog_manager.cc:4841] TS 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127): tablet f5db2c4383f0408b960b375dcd3619e8 (table test-table [id=359ef82508de4f948cc02c5b331b4684]) successfully deleted
I20260926 14:45:11.846642 8032 tablet_service.cc:1467] Processing CreateTablet for tablet ae7d6f1619c947728bc6138c5d20ff68 (DEFAULT_TABLE table=test-table [id=19ae5b20305f4e4cb6f2451b53344d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.846961 8032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7d6f1619c947728bc6138c5d20ff68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.849357 8407 tablet_bootstrap.cc:492] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3: Bootstrap starting.
I20260926 14:45:11.850245 8407 tablet_bootstrap.cc:654] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.851511 8407 tablet_bootstrap.cc:492] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3: No bootstrap required, opened a new log
I20260926 14:45:11.851590 8407 ts_tablet_manager.cc:1397] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:45:11.851616 8172 tablet_service.cc:1467] Processing CreateTablet for tablet ae7d6f1619c947728bc6138c5d20ff68 (DEFAULT_TABLE table=test-table [id=19ae5b20305f4e4cb6f2451b53344d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.851877 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7d6f1619c947728bc6138c5d20ff68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.851984 8407 raft_consensus.cc:348] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.852089 8407 raft_consensus.cc:374] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.852116 8407 raft_consensus.cc:729] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d9a124c493643b489d193521641a9e3, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.852226 8407 consensus_queue.cc:260] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.852473 8407 ts_tablet_manager.cc:1428] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:45:11.854209 8406 tablet_bootstrap.cc:492] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288: Bootstrap starting.
I20260926 14:45:11.854447 8312 tablet_service.cc:1467] Processing CreateTablet for tablet ae7d6f1619c947728bc6138c5d20ff68 (DEFAULT_TABLE table=test-table [id=19ae5b20305f4e4cb6f2451b53344d04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:11.854705 8312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae7d6f1619c947728bc6138c5d20ff68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:11.854892 8406 tablet_bootstrap.cc:654] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.855814 8406 tablet_bootstrap.cc:492] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288: No bootstrap required, opened a new log
I20260926 14:45:11.855875 8406 ts_tablet_manager.cc:1397] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 14:45:11.856164 8406 raft_consensus.cc:348] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.856245 8406 raft_consensus.cc:374] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.856266 8406 raft_consensus.cc:729] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d1abb1abc140b295d86feb95ffd288, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.856369 8406 consensus_queue.cc:260] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.856591 8406 ts_tablet_manager.cc:1428] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 14:45:11.857064 8408 tablet_bootstrap.cc:492] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c: Bootstrap starting.
I20260926 14:45:11.857919 8408 tablet_bootstrap.cc:654] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:11.859414 8408 tablet_bootstrap.cc:492] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c: No bootstrap required, opened a new log
I20260926 14:45:11.859499 8408 ts_tablet_manager.cc:1397] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 14:45:11.859902 8408 raft_consensus.cc:348] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.860010 8408 raft_consensus.cc:374] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:11.860039 8408 raft_consensus.cc:729] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac03279ec964031bb8a706d4b052b6c, State: Initialized, Role: FOLLOWER
I20260926 14:45:11.860149 8408 consensus_queue.cc:260] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.860409 8408 ts_tablet_manager.cc:1428] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 14:45:11.976907 8412 raft_consensus.cc:482] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:45:11.977053 8412 raft_consensus.cc:504] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.977456 8412 leader_election.cc:288] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317), 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127)
I20260926 14:45:11.977900 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7d6f1619c947728bc6138c5d20ff68" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9a124c493643b489d193521641a9e3" is_pre_election: true
I20260926 14:45:11.977903 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7d6f1619c947728bc6138c5d20ff68" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac03279ec964031bb8a706d4b052b6c" is_pre_election: true
I20260926 14:45:11.978042 8332 raft_consensus.cc:2454] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 0.
I20260926 14:45:11.978042 8052 raft_consensus.cc:2454] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 0.
I20260926 14:45:11.978351 8125 leader_election.cc:302] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28d1abb1abc140b295d86feb95ffd288, 8ac03279ec964031bb8a706d4b052b6c; no voters:
I20260926 14:45:11.978516 8412 raft_consensus.cc:2781] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:45:11.978574 8412 raft_consensus.cc:482] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:45:11.978621 8412 raft_consensus.cc:3037] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.979661 8412 raft_consensus.cc:504] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.979993 8412 leader_election.cc:288] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 election: Requested vote from peers 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317), 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127)
I20260926 14:45:11.980340 8052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7d6f1619c947728bc6138c5d20ff68" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d9a124c493643b489d193521641a9e3"
I20260926 14:45:11.980444 8052 raft_consensus.cc:3037] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.980399 8332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7d6f1619c947728bc6138c5d20ff68" candidate_uuid: "28d1abb1abc140b295d86feb95ffd288" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac03279ec964031bb8a706d4b052b6c"
I20260926 14:45:11.980525 8332 raft_consensus.cc:3037] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:11.981606 8052 raft_consensus.cc:2454] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 1.
I20260926 14:45:11.981607 8332 raft_consensus.cc:2454] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 28d1abb1abc140b295d86feb95ffd288 in term 1.
I20260926 14:45:11.981879 8126 leader_election.cc:302] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d9a124c493643b489d193521641a9e3, 28d1abb1abc140b295d86feb95ffd288; no voters:
I20260926 14:45:11.982048 8412 raft_consensus.cc:2781] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:11.982136 8412 raft_consensus.cc:686] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 1 LEADER]: Becoming Leader. State: Replica: 28d1abb1abc140b295d86feb95ffd288, State: Running, Role: LEADER
I20260926 14:45:11.982292 8412 consensus_queue.cc:237] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } }
I20260926 14:45:11.983855 7894 catalog_manager.cc:5495] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 reported cstate change: term changed from 0 to 1, leader changed from <none> to 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "28d1abb1abc140b295d86feb95ffd288" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "28d1abb1abc140b295d86feb95ffd288" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 } health_report { overall_health: UNKNOWN } } }
I20260926 14:45:12.027253 8402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:12.024699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45689 (local address 127.0.98.2:40339)
0926 14:45:12.024801 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:12.024804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:12.024815 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:45:12.024848 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:12.024850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:12.024886 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:12.024943 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:45:12.025164 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:12.025647 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:12.026385 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:12.026579 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:12.026645 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:12.026918 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:45:12.026978 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:45:12.027029 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:12.027102 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 14:45:12.027139 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:45:12.027259 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:12.024325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40339 (local address 127.0.98.2:45689)
0926 14:45:12.024467 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:12.024480 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:12.024554 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:12.024992 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:12.024995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:12.025002 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 14:45:12.025086 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:12.025090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:12.025726 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:12.025729 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:12.026303 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:12.026310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:12.026690 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:12.026693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:12.026795 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:12.026827 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 14:45:12.027044 (+ 217us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 14:45:12.027048 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:45:12.027091 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 14:45:12.027154 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 14:45:12.032740 8332 raft_consensus.cc:1261] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Refusing update from remote peer 28d1abb1abc140b295d86feb95ffd288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:12.033099 8412 consensus_queue.cc:1035] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac03279ec964031bb8a706d4b052b6c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:12.035238 8052 raft_consensus.cc:1261] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3 [term 1 FOLLOWER]: Refusing update from remote peer 28d1abb1abc140b295d86feb95ffd288: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:12.035724 8457 consensus_queue.cc:1035] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d9a124c493643b489d193521641a9e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:12.066004 7895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:12.066145 7895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:12.067982 7895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66eabb331e204caf88107e651612ec11: Sending DeleteTablet for 3 replicas of tablet ae7d6f1619c947728bc6138c5d20ff68
I20260926 14:45:12.068483 8171 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7d6f1619c947728bc6138c5d20ff68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54016
I20260926 14:45:12.068487 8032 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7d6f1619c947728bc6138c5d20ff68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34532
I20260926 14:45:12.068840 8432 tablet_replica.cc:330] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288: stopping tablet replica
I20260926 14:45:12.068849 8312 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7d6f1619c947728bc6138c5d20ff68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:12 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57878
I20260926 14:45:12.068958 8432 raft_consensus.cc:2229] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:45:12.069031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 7967
I20260926 14:45:12.069149 8432 raft_consensus.cc:2258] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:12.069314 8431 tablet_replica.cc:330] T ae7d6f1619c947728bc6138c5d20ff68 P 1d9a124c493643b489d193521641a9e3: stopping tablet replica
I20260926 14:45:12.069528 8430 tablet_replica.cc:330] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c: stopping tablet replica
I20260926 14:45:12.069617 8430 raft_consensus.cc:2229] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:12.069716 8430 raft_consensus.cc:2258] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:12.070485 8432 ts_tablet_manager.cc:1905] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:12.072674 8432 ts_tablet_manager.cc:1918] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:45:12.072727 8432 log.cc:1196] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/ae7d6f1619c947728bc6138c5d20ff68
I20260926 14:45:12.072896 8432 ts_tablet_manager.cc:1939] T ae7d6f1619c947728bc6138c5d20ff68 P 28d1abb1abc140b295d86feb95ffd288: Deleting consensus metadata
I20260926 14:45:12.073521 7881 catalog_manager.cc:4841] TS 28d1abb1abc140b295d86feb95ffd288 (127.0.98.2:40339): tablet ae7d6f1619c947728bc6138c5d20ff68 (table test-table [id=19ae5b20305f4e4cb6f2451b53344d04]) successfully deleted
W20260926 14:45:12.075824 7882 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34317 (error 108)
I20260926 14:45:12.076144 8430 ts_tablet_manager.cc:1905] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 14:45:12.076828 8497 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:12.076474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34317 (local address 127.0.0.1:34546)
0926 14:45:12.076660 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:12.076724 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34317: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":50,"threads_started":1}
I20260926 14:45:12.076994 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 8107
W20260926 14:45:12.077016 7882 catalog_manager.cc:4584] TS 1d9a124c493643b489d193521641a9e3 (127.0.98.1:34317): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ae7d6f1619c947728bc6138c5d20ff68: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34317: connect: Connection reset by peer (error 104)
I20260926 14:45:12.078455 8430 ts_tablet_manager.cc:1918] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 14:45:12.078537 8430 log.cc:1196] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/ae7d6f1619c947728bc6138c5d20ff68
I20260926 14:45:12.078917 8430 ts_tablet_manager.cc:1939] T ae7d6f1619c947728bc6138c5d20ff68 P 8ac03279ec964031bb8a706d4b052b6c: Deleting consensus metadata
I20260926 14:45:12.079638 7881 catalog_manager.cc:4841] TS 8ac03279ec964031bb8a706d4b052b6c (127.0.98.3:44127): tablet ae7d6f1619c947728bc6138c5d20ff68 (table test-table [id=19ae5b20305f4e4cb6f2451b53344d04]) successfully deleted
I20260926 14:45:12.084676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 8247
I20260926 14:45:12.090632 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 7863
2026-09-26T14:45:12Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3032 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:45:12 dist-test-slave-s7k6 krb5kdc[8500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:45:12 dist-test-slave-s7k6 krb5kdc[8500](info): set up 2 sockets
Sep 26 14:45:12 dist-test-slave-s7k6 krb5kdc[8500](info): commencing operation
krb5kdc: starting...
W20260926 14:45:13.039192 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:45:13 dist-test-slave-s7k6 krb5kdc[8500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:45:13.156898 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46455
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:13.264820 8513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:13.265082 8513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:13.265122 8513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:13.269202 8513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:13.269263 8513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:13.269281 8513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:13.269296 8513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:13.269311 8513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:45:13.274561 8513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:13.275583 8513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:13.276582 8513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:13.284940 8518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:13.284940 8519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:13.285128 8513 server_base.cc:1034] running on GCE node
W20260926 14:45:13.285013 8521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:13.285868 8513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:13.287014 8513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:13.288163 8513 hybrid_clock.cc:648] HybridClock initialized: now 1790433913288142 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T14:45:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1204 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:45:13 dist-test-slave-s7k6 krb5kdc[8526](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:45:13 dist-test-slave-s7k6 krb5kdc[8526](info): set up 2 sockets
Sep 26 14:45:13 dist-test-slave-s7k6 krb5kdc[8526](info): commencing operation
krb5kdc: starting...
W20260926 14:45:14.251248 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:45:14 dist-test-slave-s7k6 krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:45:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:45:14.369521 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37663
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:14.479575 8539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:14.479885 8539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:14.479925 8539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:14.484015 8539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:14.484078 8539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:14.484097 8539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:14.484112 8539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:14.484126 8539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:45:14.489517 8539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:14.490548 8539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:14.491559 8539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:14.499974 8547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:14.500061 8545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:14.499975 8544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:14.500293 8539 server_base.cc:1034] running on GCE node
I20260926 14:45:14.500840 8539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:14.502059 8539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:14.503273 8539 hybrid_clock.cc:648] HybridClock initialized: now 1790433914503244 us; error 44 us; skew 500 ppm
Sep 26 14:45:14 dist-test-slave-s7k6 krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:14.506001 8539 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:45:14.506976 8539 webserver.cc:460] Webserver started at http://127.0.98.62:39453/ using document root <none> and password file <none>
I20260926 14:45:14.507529 8539 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:14.507596 8539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:14.507797 8539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:14.509865 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "7cdcc4d67b824bad82cab913ed2b4af4"
format_stamp: "Formatted at 2026-09-26 14:45:14 on dist-test-slave-s7k6"
server_key: "e0f4315061b7fffd5a678460cac147d0"
server_key_iv: "961a33738a35f79ec32c8af895e4cb14"
server_key_version: "encryptionkey@0"
I20260926 14:45:14.510414 8539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "7cdcc4d67b824bad82cab913ed2b4af4"
format_stamp: "Formatted at 2026-09-26 14:45:14 on dist-test-slave-s7k6"
server_key: "e0f4315061b7fffd5a678460cac147d0"
server_key_iv: "961a33738a35f79ec32c8af895e4cb14"
server_key_version: "encryptionkey@0"
I20260926 14:45:14.514195 8539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:45:14.516754 8554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:14.518107 8539 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:45:14.518234 8539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "7cdcc4d67b824bad82cab913ed2b4af4"
format_stamp: "Formatted at 2026-09-26 14:45:14 on dist-test-slave-s7k6"
server_key: "e0f4315061b7fffd5a678460cac147d0"
server_key_iv: "961a33738a35f79ec32c8af895e4cb14"
server_key_version: "encryptionkey@0"
I20260926 14:45:14.518342 8539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:14.533854 8539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:14.536937 8539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:14.537119 8539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:14.543916 8539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37663
I20260926 14:45:14.543936 8606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37663 every 8 connection(s)
I20260926 14:45:14.544960 8539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:14.545368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 8539
I20260926 14:45:14.545467 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:45:14.545732 392 external_mini_cluster.cc:1428] Setting key cade1b7a4b9dd5d7704dae4ae0eb6dfa
I20260926 14:45:14.548202 8607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:14.555009 8607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4: Bootstrap starting.
Sep 26 14:45:14 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:45:14.557685 8607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:14.558539 8607 log.cc:824] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:14.560945 8607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4: No bootstrap required, opened a new log
I20260926 14:45:14.561172 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:14.547209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44778 (local address 127.0.98.62:37663)
0926 14:45:14.547680 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:14.547701 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:14.547745 (+ 44us) server_negotiation.cc:407] Connection header received
0926 14:45:14.548615 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:14.548639 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:14.549063 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:14.549439 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:14.550125 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:14.551669 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:14.552621 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:14.553296 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:14.553454 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:14.555904 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:14.555910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:14.555927 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:14.555960 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:14.557918 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:14.558420 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:14.558427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:14.558433 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:14.558589 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:14.558939 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:14.558943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:14.558945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:14.559384 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:14.559589 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:14.559859 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 14:45:14.560161 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":131,"threads_started":1}
I20260926 14:45:14.564997 8607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 14:45:14.565253 8607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:14.565290 8607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cdcc4d67b824bad82cab913ed2b4af4, State: Initialized, Role: FOLLOWER
I20260926 14:45:14.565701 8607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 14:45:14.565826 8607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:14.565864 8607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:14.565938 8607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:14.566933 8607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 14:45:14.567276 8607 leader_election.cc:302] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cdcc4d67b824bad82cab913ed2b4af4; no voters:
I20260926 14:45:14.567569 8607 leader_election.cc:288] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:45:14.567716 8612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:14.567929 8612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 1 LEADER]: Becoming Leader. State: Replica: 7cdcc4d67b824bad82cab913ed2b4af4, State: Running, Role: LEADER
I20260926 14:45:14.568280 8612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 14:45:14.568817 8607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:14.569978 8614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cdcc4d67b824bad82cab913ed2b4af4. Latest consensus state: current_term: 1 leader_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } } }
I20260926 14:45:14.570107 8614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:14.570397 8613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } } }
I20260926 14:45:14.570487 8621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:14.570600 8613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:14.573480 8621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:14.578734 8621 catalog_manager.cc:1347] Generated new cluster ID: a90725b69d5340d493a4af0d910d21d7
I20260926 14:45:14.578841 8621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:14.584364 8621 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:45:14.585603 8621 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:14.602945 8621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4: Generated new TSK 0
I20260926 14:45:14.603737 8621 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:45:14.635511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:14.747287 8633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:14.747552 8633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:14.747594 8633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:14.751654 8633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:14.751718 8633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:14.751776 8633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:45:14.757378 8633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:14.758617 8633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:14.759928 8633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:14.770393 8641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:14.770396 8638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:14.770711 8633 server_base.cc:1034] running on GCE node
W20260926 14:45:14.770413 8639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:14.771193 8633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:14.771819 8633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:14.772964 8633 hybrid_clock.cc:648] HybridClock initialized: now 1790433914772937 us; error 42 us; skew 500 ppm
Sep 26 14:45:14 dist-test-slave-s7k6 krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:14.775822 8633 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:45:14.776734 8633 webserver.cc:460] Webserver started at http://127.0.98.1:40101/ using document root <none> and password file <none>
I20260926 14:45:14.777261 8633 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:14.777318 8633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:14.777482 8633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:14.779817 8633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "1bdecfceba574fd2a285590f6a7aec07"
format_stamp: "Formatted at 2026-09-26 14:45:14 on dist-test-slave-s7k6"
server_key: "67fa65ba867cc6563a21c894bd234c32"
server_key_iv: "58a7cf29fe992a10872f7874b5c4e80a"
server_key_version: "encryptionkey@0"
I20260926 14:45:14.780478 8633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1bdecfceba574fd2a285590f6a7aec07"
format_stamp: "Formatted at 2026-09-26 14:45:14 on dist-test-slave-s7k6"
server_key: "67fa65ba867cc6563a21c894bd234c32"
server_key_iv: "58a7cf29fe992a10872f7874b5c4e80a"
server_key_version: "encryptionkey@0"
I20260926 14:45:14.784852 8633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:45:14.787405 8648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:14.788782 8633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:14.788900 8633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "1bdecfceba574fd2a285590f6a7aec07"
format_stamp: "Formatted at 2026-09-26 14:45:14 on dist-test-slave-s7k6"
server_key: "67fa65ba867cc6563a21c894bd234c32"
server_key_iv: "58a7cf29fe992a10872f7874b5c4e80a"
server_key_version: "encryptionkey@0"
I20260926 14:45:14.789022 8633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:14.851951 8633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:14.855216 8633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:14.855374 8633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:14.855926 8633 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:14.856731 8633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:14.856777 8633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:14.856817 8633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:14.856837 8633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:14.865423 8633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37177
I20260926 14:45:14.865444 8761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37177 every 8 connection(s)
I20260926 14:45:14.866400 8633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:45:14.872180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 8633
I20260926 14:45:14.872289 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:45:14.872535 392 external_mini_cluster.cc:1428] Setting key 4dd04f90ac56ec7c100be2be97096618
Sep 26 14:45:14 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433914, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:45:14.880954 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:14.868316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36317 (local address 127.0.98.62:37663)
0926 14:45:14.868446 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:14.868450 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:14.869297 (+ 847us) server_negotiation.cc:407] Connection header received
0926 14:45:14.870828 (+ 1531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:14.870831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:14.870886 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:14.870958 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:14.872090 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:14.872718 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:14.873990 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:14.874239 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:14.874326 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:14.876892 (+ 2566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:14.876897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:14.876899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:14.876927 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:14.878525 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:14.879199 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:14.879202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:14.879203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:14.879280 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:14.879696 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:14.879699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:14.879700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:14.879896 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:14.879989 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:14.880703 (+ 714us) server_negotiation.cc:299] Negotiation successful
0926 14:45:14.880825 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 14:45:14.882099 8764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:14.868697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:36317)
0926 14:45:14.869151 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:14.869193 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:14.870567 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:14.871121 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:14.871129 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:14.871472 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:14.871953 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:14.871967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:14.872829 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:14.872832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:14.873883 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:14.873894 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:14.874374 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:14.874378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:14.874542 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:14.875109 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:14.875126 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:14.876711 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:14.878661 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:14.878669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:14.878685 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:14.879093 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:14.879365 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:14.879368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:14.879370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:14.879556 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:14.879998 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:14.880005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:14.880356 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 14:45:14.880579 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 14:45:14.880951 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":108,"threads_started":1}
I20260926 14:45:14.883582 8762 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37663
I20260926 14:45:14.883898 8762 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 14:45:14.884490 8762 heartbeater.cc:511] Master 127.0.98.62:37663 requested a full tablet report, sending...
I20260926 14:45:14.886193 8571 ts_manager.cc:194] Registered new tserver with Master: 1bdecfceba574fd2a285590f6a7aec07 (127.0.98.1:37177)
I20260926 14:45:14.887400 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36317
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:45:14.899055 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:15.008850 8767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:15.009155 8767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:15.009214 8767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:15.013497 8767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:15.013576 8767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:15.013661 8767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:45:15.019287 8767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8767
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:15.020371 8767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:15.021430 8767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:15.031303 8773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:15.031450 8767 server_base.cc:1034] running on GCE node
W20260926 14:45:15.031307 8775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:15.031307 8772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:15.032124 8767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:15.032816 8767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:15.033998 8767 hybrid_clock.cc:648] HybridClock initialized: now 1790433915033975 us; error 36 us; skew 500 ppm
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:15.036921 8767 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:45:15.037845 8767 webserver.cc:460] Webserver started at http://127.0.98.2:41119/ using document root <none> and password file <none>
I20260926 14:45:15.038403 8767 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:15.038468 8767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:15.038671 8767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:15.040884 8767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "3190bb2c6f284a7d80987a0d1b4d953a"
format_stamp: "Formatted at 2026-09-26 14:45:15 on dist-test-slave-s7k6"
server_key: "d7fcdfdf2dfb550770b197b42e7a0999"
server_key_iv: "9d6a1414b8a4bf78e6b0e5ba3f64821b"
server_key_version: "encryptionkey@0"
I20260926 14:45:15.041493 8767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3190bb2c6f284a7d80987a0d1b4d953a"
format_stamp: "Formatted at 2026-09-26 14:45:15 on dist-test-slave-s7k6"
server_key: "d7fcdfdf2dfb550770b197b42e7a0999"
server_key_iv: "9d6a1414b8a4bf78e6b0e5ba3f64821b"
server_key_version: "encryptionkey@0"
I20260926 14:45:15.045472 8767 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:45:15.048286 8782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.049715 8767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:45:15.049853 8767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "3190bb2c6f284a7d80987a0d1b4d953a"
format_stamp: "Formatted at 2026-09-26 14:45:15 on dist-test-slave-s7k6"
server_key: "d7fcdfdf2dfb550770b197b42e7a0999"
server_key_iv: "9d6a1414b8a4bf78e6b0e5ba3f64821b"
server_key_version: "encryptionkey@0"
I20260926 14:45:15.049963 8767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:15.059314 8767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:15.062439 8767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:15.062628 8767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:15.063220 8767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:15.064024 8767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:15.064076 8767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.064117 8767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:15.064146 8767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.072567 8767 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38705
I20260926 14:45:15.072695 8895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38705 every 8 connection(s)
I20260926 14:45:15.073477 8767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:45:15.074846 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 8767
I20260926 14:45:15.074949 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:45:15.075205 392 external_mini_cluster.cc:1428] Setting key fdd6f5f507d17f2d5a9bbd9e045023b3
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:45:15.087791 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:15.075347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59211 (local address 127.0.98.62:37663)
0926 14:45:15.075489 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:15.075494 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:15.076445 (+ 951us) server_negotiation.cc:407] Connection header received
0926 14:45:15.077842 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:15.077846 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:15.077900 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:15.077969 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:15.079196 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.079732 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.080907 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.081141 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.081233 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.083894 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:15.083897 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:15.083900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:15.083927 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:15.085512 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:15.086075 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:15.086080 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:15.086082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:15.086168 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:15.086556 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:15.086559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:15.086561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:15.086860 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:15.086948 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:15.087485 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 14:45:15.087620 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:45:15.088652 8898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:15.075795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:59211)
0926 14:45:15.076295 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:15.076354 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:15.077637 (+ 1283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:15.078148 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:15.078156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:15.078512 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:15.079057 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.079072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.079854 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.079857 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.080777 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.080788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.081291 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.081296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.081475 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.082160 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:15.082178 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:15.083710 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:15.085641 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:15.085667 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:15.085686 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:15.085964 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:15.086268 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:15.086271 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:15.086274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:15.086465 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:15.086958 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:15.086965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:15.087248 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 14:45:15.087426 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 14:45:15.087786 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":103,"threads_started":1}
I20260926 14:45:15.089916 8896 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37663
I20260926 14:45:15.090167 8896 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:15.090659 8896 heartbeater.cc:511] Master 127.0.98.62:37663 requested a full tablet report, sending...
I20260926 14:45:15.091777 8571 ts_manager.cc:194] Registered new tserver with Master: 3190bb2c6f284a7d80987a0d1b4d953a (127.0.98.2:38705)
I20260926 14:45:15.092556 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59211
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:45:15.101646 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:15.222172 8901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:15.222474 8901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:15.222555 8901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:15.226791 8901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:15.226862 8901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:15.226944 8901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:45:15.232384 8901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:15.233443 8901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:15.234478 8901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:15.244439 8909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:15.244515 8901 server_base.cc:1034] running on GCE node
W20260926 14:45:15.244454 8906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:15.244463 8907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:15.245189 8901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:15.245903 8901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:15.247066 8901 hybrid_clock.cc:648] HybridClock initialized: now 1790433915247042 us; error 39 us; skew 500 ppm
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:15.249888 8901 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:45:15.250847 8901 webserver.cc:460] Webserver started at http://127.0.98.3:46055/ using document root <none> and password file <none>
I20260926 14:45:15.251386 8901 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:15.251444 8901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:15.251612 8901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:15.253787 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "37d308153c5d4c6bbe79aeef7619bfaf"
format_stamp: "Formatted at 2026-09-26 14:45:15 on dist-test-slave-s7k6"
server_key: "c3bca582533ce99906221015a896ada3"
server_key_iv: "ae162f68710496d5e887f915dc215342"
server_key_version: "encryptionkey@0"
I20260926 14:45:15.254346 8901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "37d308153c5d4c6bbe79aeef7619bfaf"
format_stamp: "Formatted at 2026-09-26 14:45:15 on dist-test-slave-s7k6"
server_key: "c3bca582533ce99906221015a896ada3"
server_key_iv: "ae162f68710496d5e887f915dc215342"
server_key_version: "encryptionkey@0"
I20260926 14:45:15.258112 8901 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:45:15.260504 8916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.261725 8901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 14:45:15.261871 8901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "37d308153c5d4c6bbe79aeef7619bfaf"
format_stamp: "Formatted at 2026-09-26 14:45:15 on dist-test-slave-s7k6"
server_key: "c3bca582533ce99906221015a896ada3"
server_key_iv: "ae162f68710496d5e887f915dc215342"
server_key_version: "encryptionkey@0"
I20260926 14:45:15.261970 8901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:15.285542 8901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:15.288872 8901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:15.289052 8901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:15.289671 8901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:15.290539 8901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:15.290585 8901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.290624 8901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:15.290645 8901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.299245 8901 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33073
I20260926 14:45:15.299275 9029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33073 every 8 connection(s)
I20260926 14:45:15.300134 8901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:45:15.308077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 8901
I20260926 14:45:15.308238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:45:15.308563 392 external_mini_cluster.cc:1428] Setting key e9968fa87916c3b32c083a3f82bc8789
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:45:15.314254 8610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:15.301933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48761 (local address 127.0.98.62:37663)
0926 14:45:15.302073 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:15.302078 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:15.303080 (+ 1002us) server_negotiation.cc:407] Connection header received
0926 14:45:15.304494 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:15.304498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:15.304547 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:15.304624 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:15.305851 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.306386 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.307576 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.307802 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.307885 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.310776 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:15.310781 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:15.310784 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:15.310813 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:15.312170 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:15.312758 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:15.312761 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:15.312763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:15.312864 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:15.313212 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:15.313215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:15.313216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:15.313407 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:15.313483 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:15.314018 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 14:45:15.314128 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:45:15.315173 9032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:15.302403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:48761)
0926 14:45:15.302932 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:15.302978 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:15.304273 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:15.304836 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:15.304844 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:15.305210 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:15.305707 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.305721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.306501 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.306504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.307461 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.307471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.307973 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.307977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.308221 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.308866 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:15.308888 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:15.310592 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:15.312287 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:15.312292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:15.312306 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:15.312573 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:15.312959 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:15.312962 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:15.312963 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:15.313119 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:15.313515 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:15.313520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:15.313780 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 14:45:15.313949 (+ 169us) client_negotiation.cc:240] Negotiation successful
0926 14:45:15.314219 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":128,"threads_started":1}
I20260926 14:45:15.316315 9030 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37663
I20260926 14:45:15.316537 9030 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:15.317011 9030 heartbeater.cc:511] Master 127.0.98.62:37663 requested a full tablet report, sending...
I20260926 14:45:15.318097 8571 ts_manager.cc:194] Registered new tserver with Master: 37d308153c5d4c6bbe79aeef7619bfaf (127.0.98.3:33073)
I20260926 14:45:15.318694 8571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48761
I20260926 14:45:15.322809 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 14:45:15.391010 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 8633
I20260926 14:45:15.395753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 8767
I20260926 14:45:15.400563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 8901
I20260926 14:45:15.405196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 8539
I20260926 14:45:15.410621 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37663
--webserver_interface=127.0.98.62
--webserver_port=39453
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:15.520514 9039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:15.520828 9039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:15.520884 9039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:15.525107 9039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:15.525188 9039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:15.525215 9039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:15.525238 9039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:15.525256 9039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:45:15.530819 9039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39453
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:15.531876 9039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:15.532884 9039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:15.541213 9044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:15.541172 9047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:15.541283 9045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:15.541388 9039 server_base.cc:1034] running on GCE node
I20260926 14:45:15.542140 9039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:15.543386 9039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:15.544553 9039 hybrid_clock.cc:648] HybridClock initialized: now 1790433915544528 us; error 39 us; skew 500 ppm
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:15.547263 9039 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:45:15.548203 9039 webserver.cc:460] Webserver started at http://127.0.98.62:39453/ using document root <none> and password file <none>
I20260926 14:45:15.548787 9039 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:15.548872 9039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:15.552906 9039 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:45:15.555179 9054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.556456 9039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:45:15.556571 9039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "7cdcc4d67b824bad82cab913ed2b4af4"
format_stamp: "Formatted at 2026-09-26 14:45:14 on dist-test-slave-s7k6"
server_key: "e0f4315061b7fffd5a678460cac147d0"
server_key_iv: "961a33738a35f79ec32c8af895e4cb14"
server_key_version: "encryptionkey@0"
I20260926 14:45:15.557122 9039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:15.572667 9039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:15.576465 9039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:15.576682 9039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:15.583927 9039 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37663
I20260926 14:45:15.583954 9106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37663 every 8 connection(s)
I20260926 14:45:15.585055 9039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:15.586494 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 9039
I20260926 14:45:15.587009 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37177
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37663
--webserver_port=40101
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:45:15.590072 9107 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:45:15.591809 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4: Bootstrap starting.
I20260926 14:45:15.608361 9107 log.cc:824] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:15.615093 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:45:15.615455 9107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4: Bootstrap complete.
I20260926 14:45:15.619539 9107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 14:45:15.619926 9107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cdcc4d67b824bad82cab913ed2b4af4, State: Initialized, Role: FOLLOWER
I20260926 14:45:15.620388 9107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 14:45:15.620505 9107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:15.620569 9107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:15.620658 9107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:45:15.622646 9107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 14:45:15.622985 9107 leader_election.cc:302] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cdcc4d67b824bad82cab913ed2b4af4; no voters:
I20260926 14:45:15.623318 9107 leader_election.cc:288] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:45:15.623420 9111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:45:15.623601 9111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [term 2 LEADER]: Becoming Leader. State: Replica: 7cdcc4d67b824bad82cab913ed2b4af4, State: Running, Role: LEADER
I20260926 14:45:15.623904 9111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } }
I20260926 14:45:15.624555 9107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:15.625234 9112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } } }
I20260926 14:45:15.625550 9112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:15.625468 9111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cdcc4d67b824bad82cab913ed2b4af4. Latest consensus state: current_term: 2 leader_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cdcc4d67b824bad82cab913ed2b4af4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37663 } } }
I20260926 14:45:15.625859 9111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:15.626434 9120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:15.629112 9120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:15.629658 9120 catalog_manager.cc:1259] Loaded cluster ID: a90725b69d5340d493a4af0d910d21d7
I20260926 14:45:15.629707 9120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:15.631425 9120 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:15.631960 9120 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7cdcc4d67b824bad82cab913ed2b4af4: Loaded TSK: 0
I20260926 14:45:15.632889 9120 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:45:15.705487 9109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:15.705766 9109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:15.705837 9109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:15.709888 9109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:15.709966 9109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:15.710049 9109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:45:15.715583 9109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37177
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40101
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:15.716730 9109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:15.717736 9109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:15.728382 9133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:15.728413 9136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:15.728432 9134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:15.728860 9109 server_base.cc:1034] running on GCE node
I20260926 14:45:15.729282 9109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:15.729938 9109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:15.731102 9109 hybrid_clock.cc:648] HybridClock initialized: now 1790433915731089 us; error 30 us; skew 500 ppm
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433915, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:15.733989 9109 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 14:45:15.735003 9109 webserver.cc:460] Webserver started at http://127.0.98.1:40101/ using document root <none> and password file <none>
I20260926 14:45:15.735551 9109 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:15.735634 9109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:15.739576 9109 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:45:15.741647 9143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.743158 9109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:15.743276 9109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "1bdecfceba574fd2a285590f6a7aec07"
format_stamp: "Formatted at 2026-09-26 14:45:14 on dist-test-slave-s7k6"
server_key: "67fa65ba867cc6563a21c894bd234c32"
server_key_iv: "58a7cf29fe992a10872f7874b5c4e80a"
server_key_version: "encryptionkey@0"
I20260926 14:45:15.743810 9109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:15.786657 9109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:15.789938 9109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:15.790131 9109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:15.790748 9109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:15.791642 9109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:15.791699 9109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.791754 9109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:15.791782 9109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.800424 9109 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37177
I20260926 14:45:15.800459 9256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37177 every 8 connection(s)
I20260926 14:45:15.801457 9109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:45:15.803886 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 9109
I20260926 14:45:15.804381 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:38705
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37663
--webserver_port=41119
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:45:15.816956 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:15.803671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42827 (local address 127.0.98.62:37663)
0926 14:45:15.804110 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:15.804130 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:15.804696 (+ 566us) server_negotiation.cc:407] Connection header received
0926 14:45:15.806388 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:15.806401 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:15.806735 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:15.807094 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:15.808635 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.809622 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.811394 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.811909 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.812048 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.816181 (+ 4133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42827: BlockingRecv error: recv got EOF from 127.0.98.1:42827 (error 108)
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":101,"threads_started":1}
I20260926 14:45:15.816964 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:15.803811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:42827)
0926 14:45:15.804337 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:15.804381 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:15.805890 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:15.807243 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:15.807254 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:15.807750 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:15.808442 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.808465 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.809778 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.809782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.811260 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.811274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.812086 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.812091 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.812307 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.813036 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:15.813060 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:15.815910 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":141,"threads_started":1}
W20260926 14:45:15.817540 9257 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37663 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37663: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:15.823101 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:15.818037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:39841)
0926 14:45:15.818204 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:15.818218 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:15.818320 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:15.818543 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:15.818546 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:15.818700 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:15.818826 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.818834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.819564 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.819567 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.820127 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.820135 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.820569 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.820572 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.820661 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.821127 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:15.821138 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:15.822971 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 14:45:15.823217 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:15.818095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39841 (local address 127.0.98.62:37663)
0926 14:45:15.818205 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:15.818209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:15.818285 (+ 76us) server_negotiation.cc:407] Connection header received
0926 14:45:15.818406 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:15.818409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:15.818454 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:15.818518 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:15.818947 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.819465 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.820251 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.820471 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.820557 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.823122 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39841: BlockingRecv error: recv got EOF from 127.0.98.1:39841 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:15.828853 9260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:15.823806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:34751)
0926 14:45:15.823909 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:15.823923 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:15.823993 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:15.824219 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:15.824221 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:15.824358 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:15.824431 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.824436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.825326 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.825329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.825880 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.825889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.826383 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.826386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.826443 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.826937 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:15.826948 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:15.828728 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 14:45:15.828984 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:15.823884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34751 (local address 127.0.98.62:37663)
0926 14:45:15.824009 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:15.824013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:15.824044 (+ 31us) server_negotiation.cc:407] Connection header received
0926 14:45:15.824082 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:15.824085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:15.824128 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:15.824207 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:15.824522 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.825230 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.825975 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.826285 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.826379 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.828874 (+ 2495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34751: BlockingRecv error: recv got EOF from 127.0.98.1:34751 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 14:45:15.829110 9257 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:45:15.919504 9261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:15.919804 9261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:15.919868 9261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:15.924046 9261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:15.924115 9261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:15.924196 9261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:45:15.929710 9261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:38705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41119
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:15.930860 9261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:15.931916 9261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:15.941489 9266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:15.941526 9269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:15.941578 9261 server_base.cc:1034] running on GCE node
W20260926 14:45:15.941484 9267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:15.942157 9261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:15.942900 9261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:15.944067 9261 hybrid_clock.cc:648] HybridClock initialized: now 1790433915944037 us; error 50 us; skew 500 ppm
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433915, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:15.947057 9261 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 14:45:15.947981 9261 webserver.cc:460] Webserver started at http://127.0.98.2:41119/ using document root <none> and password file <none>
I20260926 14:45:15.948534 9261 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:15.948609 9261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:15.952642 9261 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:45:15.954970 9276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.956262 9261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:45:15.956382 9261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "3190bb2c6f284a7d80987a0d1b4d953a"
format_stamp: "Formatted at 2026-09-26 14:45:15 on dist-test-slave-s7k6"
server_key: "d7fcdfdf2dfb550770b197b42e7a0999"
server_key_iv: "9d6a1414b8a4bf78e6b0e5ba3f64821b"
server_key_version: "encryptionkey@0"
I20260926 14:45:15.956965 9261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:15.969579 9261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:15.972637 9261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:15.972816 9261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:15.973404 9261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:15.974205 9261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:15.974249 9261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.974289 9261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:15.974304 9261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:15.982929 9261 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38705
I20260926 14:45:15.982975 9389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38705 every 8 connection(s)
I20260926 14:45:15.983973 9261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:45:15.991070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 9261
I20260926 14:45:15.991523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33073
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37663
--webserver_port=46055
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:45:15.995066 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:15.985894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45987 (local address 127.0.98.62:37663)
0926 14:45:15.986041 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:15.986045 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:15.986812 (+ 767us) server_negotiation.cc:407] Connection header received
0926 14:45:15.988163 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:15.988166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:15.988232 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:15.988305 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:15.989517 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.990035 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.991374 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.991606 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.991722 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.994934 (+ 3212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45987: BlockingRecv error: recv got EOF from 127.0.98.2:45987 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:45:15.995558 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:15.986188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:45987)
0926 14:45:15.986650 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:15.986694 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:15.987938 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:15.988520 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:15.988527 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:15.988881 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:15.989366 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.989382 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.990151 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.990154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.991235 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.991246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.991799 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.991803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.991977 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.992554 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:15.992570 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:15.994737 (+ 2167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":113,"threads_started":1}
W20260926 14:45:15.995998 9390 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37663 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37663: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:15 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:16.001065 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:15.996402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:47895)
0926 14:45:15.996491 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:15.996504 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:15.996576 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:15.996795 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:15.996798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:15.996943 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:15.997028 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.997033 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.997709 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.997713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.998253 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:15.998261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.998660 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.998663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:15.998748 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:15.999244 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:15.999255 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:16.000935 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 14:45:16.001166 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:15.996502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47895 (local address 127.0.98.62:37663)
0926 14:45:15.996620 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:15.996623 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:15.996636 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:45:15.996674 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:15.996676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:15.996709 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:15.996777 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:15.997128 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.997614 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.998364 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:15.998569 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:15.998661 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.001061 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47895: BlockingRecv error: recv got EOF from 127.0.98.2:47895 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 14:45:16 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:16 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:16.006678 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:16.001672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:34517)
0926 14:45:16.001770 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:16.001784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:16.001858 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:16.002066 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:16.002068 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:16.002188 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:16.002262 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:16.002292 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.003330 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.003333 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:16.003864 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:16.003872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.004407 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.004410 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:16.004467 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.004950 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:16.004962 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:16.006565 (+ 1603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260926 14:45:16.006889 9390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:45:16.007084 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:16.001764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34517 (local address 127.0.98.62:37663)
0926 14:45:16.001875 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:16.001879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:16.001892 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:45:16.001931 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:16.001933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:16.001973 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:16.002054 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:16.002382 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.003215 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.003965 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.004303 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.004401 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.006717 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34517: BlockingRecv error: recv got EOF from 127.0.98.2:34517 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 14:45:16.103502 9393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:16.103777 9393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:16.103819 9393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:16.107765 9393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:16.107822 9393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:16.107930 9393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:45:16.113587 9393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33073
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=46055
--tserver_master_addrs=127.0.98.62:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:16.114679 9393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:16.115746 9393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:16.125495 9399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:16.125499 9401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:16.125537 9393 server_base.cc:1034] running on GCE node
W20260926 14:45:16.125499 9398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:16.126224 9393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:16.126974 9393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:16.128131 9393 hybrid_clock.cc:648] HybridClock initialized: now 1790433916128110 us; error 36 us; skew 500 ppm
Sep 26 14:45:16 dist-test-slave-s7k6 krb5kdc[8526](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433916, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:16.139259 9393 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 14:45:16.140257 9393 webserver.cc:460] Webserver started at http://127.0.98.3:46055/ using document root <none> and password file <none>
I20260926 14:45:16.140841 9393 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:16.140925 9393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:16.144927 9393 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:45:16.147027 9408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:16.148557 9393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:45:16.148675 9393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "37d308153c5d4c6bbe79aeef7619bfaf"
format_stamp: "Formatted at 2026-09-26 14:45:15 on dist-test-slave-s7k6"
server_key: "c3bca582533ce99906221015a896ada3"
server_key_iv: "ae162f68710496d5e887f915dc215342"
server_key_version: "encryptionkey@0"
I20260926 14:45:16.149272 9393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:16.171033 9393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:16.174286 9393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:16.174472 9393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:16.175113 9393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:16.175935 9393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:16.175988 9393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:16.176043 9393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:16.176066 9393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:16.184804 9393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33073
I20260926 14:45:16.184831 9521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33073 every 8 connection(s)
I20260926 14:45:16.185835 9393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:45:16.187804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 9393
Sep 26 14:45:16 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:16 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 14:45:16.197134 9259 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:16.187576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40399 (local address 127.0.98.62:37663)
0926 14:45:16.187732 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:16.187737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:16.188619 (+ 882us) server_negotiation.cc:407] Connection header received
0926 14:45:16.190106 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:16.190109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:16.190158 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:16.190228 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:16.191434 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.191998 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.193257 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.193468 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.193544 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.197001 (+ 3457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40399: BlockingRecv error: recv got EOF from 127.0.98.3:40399 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:45:16.197430 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:16.188943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.0.98.62:37663)
0926 14:45:16.189168 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:16.189172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:16.189182 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:45:16.189218 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:16.189220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:16.189256 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:16.189334 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:16.190075 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.190601 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.191519 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.191756 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.191843 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.193279 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:16.193285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:16.193300 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:16.193330 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:16.195189 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:16.195626 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:16.195632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:16.195637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:16.195758 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:16.196110 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:16.196113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:16.196115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:16.196603 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:16.196810 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:16.197034 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 14:45:16.197288 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260926 14:45:16.197901 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:16.188001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:40399)
0926 14:45:16.188471 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:16.188522 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:16.189900 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:16.190401 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:16.190409 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:16.190746 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:16.191284 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:16.191298 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.192111 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.192113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:16.193112 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:16.193122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.193618 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.193623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:16.193788 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.194227 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:16.194247 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:16.196871 (+ 2624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":107,"threads_started":1}
W20260926 14:45:16.198400 9522 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37663 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:37663: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 14:45:16 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:16 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:16.203605 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:16.198813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:52229)
0926 14:45:16.198901 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:16.198914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:16.198981 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:16.199182 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:16.199185 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:16.199338 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:16.199447 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:16.199454 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.200254 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.200258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:16.200871 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:16.200880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.201283 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.201286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:16.201382 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.201815 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:16.201829 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:16.203506 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 14:45:16.203735 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:16.198877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52229 (local address 127.0.98.62:37663)
0926 14:45:16.198975 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:16.198979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:16.198992 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:45:16.199042 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:16.199045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:16.199084 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:16.199162 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:16.199557 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.200149 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.200979 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.201191 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.201271 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.203631 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52229: BlockingRecv error: recv got EOF from 127.0.98.3:52229 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 14:45:16 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:16 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:16.208531 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:16.204092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:46741)
0926 14:45:16.204201 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:16.204212 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:16.204274 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:16.204467 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:16.204469 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:16.204582 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:16.204652 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:16.204656 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.205379 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.205381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:16.205889 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:16.205896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.206287 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.206290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:16.206342 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.206827 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:16.206842 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:16.208436 (+ 1594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:45:16.208654 9526 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:16.204169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46741 (local address 127.0.98.62:37663)
0926 14:45:16.204289 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:16.204292 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:16.204303 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:45:16.204353 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:16.204355 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:16.204386 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:16.204444 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:16.204766 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.205274 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.205989 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.206194 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.206275 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.208548 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46741: BlockingRecv error: recv got EOF from 127.0.98.3:46741 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 14:45:16.208729 9522 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 14:45:16 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:16 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:16.834939 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:16.829674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:58711)
0926 14:45:16.829949 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:16.829966 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:16.830081 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:16.830334 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:16.830336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:16.830526 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:16.830623 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:16.830629 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.831474 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.831482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:16.832043 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:16.832051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.832486 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.832489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:16.832550 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.833069 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:16.833081 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:16.834787 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
W20260926 14:45:16.835041 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:16.829718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58711 (local address 127.0.98.62:37663)
0926 14:45:16.829958 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:16.829963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:16.830025 (+ 62us) server_negotiation.cc:407] Connection header received
0926 14:45:16.830161 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:16.830163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:16.830210 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:16.830296 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:16.830794 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.831372 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.832152 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:16.832397 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:16.832488 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:16.834921 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58711: BlockingRecv error: recv got EOF from 127.0.98.1:58711 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":70,"threads_started":1}
Sep 26 14:45:17 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:17 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:17.012881 9529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:17.007536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:36555)
0926 14:45:17.007804 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:17.007821 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:17.007942 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:17.008189 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:17.008192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:17.008396 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:17.008488 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:17.008494 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.009245 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.009249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:17.009874 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:17.009884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.010323 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.010327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:17.010407 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:17.010903 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:17.010915 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:17.012712 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
W20260926 14:45:17.013041 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:17.007651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36555 (local address 127.0.98.62:37663)
0926 14:45:17.007814 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:17.007819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:17.007896 (+ 77us) server_negotiation.cc:407] Connection header received
0926 14:45:17.008040 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:17.008044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:17.008087 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:17.008154 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:17.008608 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.009138 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.010000 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.010219 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.010303 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:17.012936 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36555: BlockingRecv error: recv got EOF from 127.0.98.2:36555 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 14:45:17 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:17 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:17.214948 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:17.209375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:49215)
0926 14:45:17.209642 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:17.209657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:17.209773 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:17.210020 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:17.210023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:17.210208 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:17.210301 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:17.210307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.211114 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.211117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:17.211706 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:17.211715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.212150 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.212153 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:17.212229 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:17.212857 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:17.212872 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:17.214820 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
W20260926 14:45:17.215044 9528 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:17.209484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49215 (local address 127.0.98.62:37663)
0926 14:45:17.209643 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:17.209647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:17.209731 (+ 84us) server_negotiation.cc:407] Connection header received
0926 14:45:17.209860 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:17.209863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:17.209922 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:17.209988 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:17.210418 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.211003 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.211813 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.212048 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.212136 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:17.214938 (+ 2802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49215: BlockingRecv error: recv got EOF from 127.0.98.3:49215 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 26 14:45:17 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:17 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:17.840760 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:17.835639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:37749)
0926 14:45:17.835956 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:17.835969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:17.836065 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:17.836280 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:17.836282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:17.836455 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:17.836550 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:17.836555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.837297 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.837300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:17.837869 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:17.837877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.838296 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.838298 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:17.838354 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:17.838833 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:17.838844 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:17.840621 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":112,"threads_started":1}
W20260926 14:45:17.840889 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:17.835732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37749 (local address 127.0.98.62:37663)
0926 14:45:17.835987 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:17.835990 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:17.836022 (+ 32us) server_negotiation.cc:407] Connection header received
0926 14:45:17.836138 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:17.836140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:17.836181 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:17.836256 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:17.836637 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.837172 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.837976 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:17.838210 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:17.838286 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:17.840772 (+ 2486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37749: BlockingRecv error: recv got EOF from 127.0.98.1:37749 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":83,"threads_started":1}
Sep 26 14:45:18 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:18 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:18.019037 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:18.013723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:57487)
0926 14:45:18.014068 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:18.014082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:18.014200 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:18.014445 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:18.014448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:18.014653 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:18.014745 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:18.014780 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.015496 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.015499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:18.016059 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:18.016066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.016464 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.016467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:18.016542 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:18.017008 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:18.017019 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:18.018863 (+ 1844us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":103,"threads_started":1}
W20260926 14:45:18.019164 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:18.013854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57487 (local address 127.0.98.62:37663)
0926 14:45:18.014021 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:18.014026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:18.014155 (+ 129us) server_negotiation.cc:407] Connection header received
0926 14:45:18.014289 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:18.014291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:18.014334 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:18.014400 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:18.014883 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.015390 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.016165 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.016371 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.016459 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:18.019051 (+ 2592us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57487: BlockingRecv error: recv got EOF from 127.0.98.2:57487 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 26 14:45:18 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:18 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:18.221249 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:18.215688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:39917)
0926 14:45:18.215997 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:18.216012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:18.216119 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:18.216357 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:18.216359 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:18.216535 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:18.216632 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:18.216638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.217431 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.217434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:18.218033 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:18.218043 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.218690 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.218693 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:18.218803 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:18.219361 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:18.219376 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:18.221113 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":74,"threads_started":1}
W20260926 14:45:18.221343 9532 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:18.215862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39917 (local address 127.0.98.62:37663)
0926 14:45:18.215984 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:18.215989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:18.216109 (+ 120us) server_negotiation.cc:407] Connection header received
0926 14:45:18.216189 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:18.216192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:18.216241 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:18.216316 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:18.216756 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.217301 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.218158 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.218382 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.218466 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:18.221247 (+ 2781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39917: BlockingRecv error: recv got EOF from 127.0.98.3:39917 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 14:45:18 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:18 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:18.846803 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:18.841546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:51719)
0926 14:45:18.841862 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:18.841878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:18.841982 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:18.842237 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:18.842239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:18.842419 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:18.842520 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:18.842526 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.843277 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.843280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:18.843852 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:18.843861 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.844252 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.844255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:18.844322 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:18.844806 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:18.844818 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:18.846628 (+ 1810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":103,"threads_started":1}
W20260926 14:45:18.846906 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:18.841641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51719 (local address 127.0.98.62:37663)
0926 14:45:18.841861 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:18.841866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:18.841949 (+ 83us) server_negotiation.cc:407] Connection header received
0926 14:45:18.842074 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:18.842077 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:18.842132 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:18.842212 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:18.842608 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.843189 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.843943 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:18.844160 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:18.844231 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:18.846790 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51719: BlockingRecv error: recv got EOF from 127.0.98.1:51719 (error 108)
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":55,"threads_started":1}
Sep 26 14:45:19 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:19 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:19.024904 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:19.019825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:54169)
0926 14:45:19.020099 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:19.020113 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:19.020214 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:19.020425 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:19.020427 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:19.020620 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:19.020717 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:19.020722 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.021414 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.021417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:19.022046 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:19.022056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.022424 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.022427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:19.022492 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:19.023059 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:19.023073 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:19.024747 (+ 1674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":68,"threads_started":1}
W20260926 14:45:19.025065 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:19.019918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54169 (local address 127.0.98.62:37663)
0926 14:45:19.020079 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:19.020083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:19.020182 (+ 99us) server_negotiation.cc:407] Connection header received
0926 14:45:19.020285 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:19.020287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:19.020333 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:19.020398 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:19.020802 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.021316 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.022135 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.022337 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.022403 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:19.024919 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54169: BlockingRecv error: recv got EOF from 127.0.98.2:54169 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 14:45:19 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:19 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:19.227577 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:19.221991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:38857)
0926 14:45:19.222301 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:19.222315 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:19.222428 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:19.222695 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:19.222698 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:19.222907 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:19.222997 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:19.223003 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.223747 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.223751 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:19.224362 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:19.224373 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.224794 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.224798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:19.224873 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:19.225381 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:19.225392 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:19.227407 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":105,"threads_started":1}
W20260926 14:45:19.227639 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:19.222091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38857 (local address 127.0.98.62:37663)
0926 14:45:19.222268 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:19.222273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:19.222385 (+ 112us) server_negotiation.cc:407] Connection header received
0926 14:45:19.222516 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:19.222519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:19.222570 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:19.222635 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:19.223120 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.223643 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.224475 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.224698 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.224781 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:19.227546 (+ 2765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38857: BlockingRecv error: recv got EOF from 127.0.98.3:38857 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 14:45:19 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:19 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:19.853085 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:19.847610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:45671)
0926 14:45:19.847888 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:19.847902 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:19.847997 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:19.848237 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:19.848239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:19.848412 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:19.848507 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:19.848513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.849283 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.849286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:19.849875 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:19.849885 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.850318 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.850322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:19.850393 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:19.850923 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:19.850935 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:19.852935 (+ 2000us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":97,"threads_started":1}
W20260926 14:45:19.853164 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:19.847719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45671 (local address 127.0.98.62:37663)
0926 14:45:19.847945 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:19.847949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:19.847964 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:45:19.848077 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:19.848079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:19.848122 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:19.848205 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:19.848603 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.849180 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.849972 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:19.850198 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:19.850280 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:19.853055 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45671: BlockingRecv error: recv got EOF from 127.0.98.1:45671 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":77,"threads_started":1}
Sep 26 14:45:20 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:20 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:20.031574 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:20.025662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:56339)
0926 14:45:20.025955 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:20.025970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:20.026092 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:20.026380 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:20.026383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:20.026575 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:20.026671 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:20.026677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.027598 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.027601 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:20.028403 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:20.028413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.028853 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.028856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:20.028926 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:20.029460 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:20.029472 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:20.031442 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":97,"threads_started":1}
W20260926 14:45:20.031713 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:20.025794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56339 (local address 127.0.98.62:37663)
0926 14:45:20.025959 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:20.025964 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:20.026043 (+ 79us) server_negotiation.cc:407] Connection header received
0926 14:45:20.026190 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:20.026193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:20.026261 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:20.026344 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:20.026815 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.027467 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.028506 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.028725 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.028807 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:20.031591 (+ 2784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56339: BlockingRecv error: recv got EOF from 127.0.98.2:56339 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 14:45:20 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:20 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:20.233378 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:20.228290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:38721)
0926 14:45:20.228554 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:20.228566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:20.228660 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:20.228896 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:20.228898 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:20.229079 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:20.229176 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:20.229181 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.229898 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.229901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:20.230493 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:20.230502 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.230918 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.230921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:20.230982 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:20.231456 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:20.231466 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:20.233215 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":101,"threads_started":1}
W20260926 14:45:20.233466 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:20.228349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38721 (local address 127.0.98.62:37663)
0926 14:45:20.228477 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:20.228482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:20.228661 (+ 179us) server_negotiation.cc:407] Connection header received
0926 14:45:20.228723 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:20.228725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:20.228778 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:20.228871 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:20.229262 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.229772 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.230586 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.230840 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.230914 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:20.233365 (+ 2451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38721: BlockingRecv error: recv got EOF from 127.0.98.3:38721 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 26 14:45:20 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:20 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:20.858914 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:20.853756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:50683)
0926 14:45:20.854022 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:20.854035 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:20.854120 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:20.854446 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:20.854447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:20.854619 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:20.854714 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:20.854720 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.855497 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.855500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:20.856088 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:20.856095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.856573 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.856576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:20.856632 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:20.857104 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:20.857116 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:20.858776 (+ 1660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":88,"threads_started":1}
W20260926 14:45:20.859026 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:20.853836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50683 (local address 127.0.98.62:37663)
0926 14:45:20.854063 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:20.854066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:20.854083 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:45:20.854235 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:20.854238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:20.854335 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:20.854419 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:20.854837 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.855410 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.856207 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:20.856480 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:20.856564 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:20.858916 (+ 2352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50683: BlockingRecv error: recv got EOF from 127.0.98.1:50683 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":55,"threads_started":1}
Sep 26 14:45:21 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:21 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:21.037765 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:21.032437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:49225)
0926 14:45:21.032713 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:21.032727 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:21.032821 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:21.033060 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:21.033062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:21.033232 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:21.033325 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:21.033330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.034034 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.034037 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:21.034614 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:21.034622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.035110 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.035114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:21.035180 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:21.035689 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:21.035703 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:21.037616 (+ 1913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
W20260926 14:45:21.037850 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:21.032587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49225 (local address 127.0.98.62:37663)
0926 14:45:21.032713 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:21.032717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:21.032778 (+ 61us) server_negotiation.cc:407] Connection header received
0926 14:45:21.032915 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:21.032917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:21.032958 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:21.033032 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:21.033412 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.033936 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.034722 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.034992 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.035072 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:21.037755 (+ 2683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49225: BlockingRecv error: recv got EOF from 127.0.98.2:49225 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 14:45:21 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:21 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:21.246039 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:21.234170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:56735)
0926 14:45:21.234460 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:21.234475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:21.234581 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:21.234835 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:21.234838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:21.235048 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:21.235142 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:21.235147 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.235870 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.235874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:21.236456 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:21.236463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.236887 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.236890 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:21.236978 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:21.243706 (+ 6728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:21.243721 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:21.245854 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":100,"threads_started":1}
W20260926 14:45:21.246097 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:21.234221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56735 (local address 127.0.98.62:37663)
0926 14:45:21.234364 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:21.234368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:21.234536 (+ 168us) server_negotiation.cc:407] Connection header received
0926 14:45:21.234677 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:21.234680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:21.234727 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:21.234816 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:21.235248 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.235771 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.236566 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.236789 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.236873 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:21.246000 (+ 9127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56735: BlockingRecv error: recv got EOF from 127.0.98.3:56735 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 26 14:45:21 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:21 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:21.865345 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:21.859645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:37487)
0926 14:45:21.859939 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:21.859956 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:21.860066 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:21.860291 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:21.860293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:21.860469 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:21.860566 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:21.860571 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.861436 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.861439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:21.862028 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:21.862038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.862499 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.862502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:21.862576 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:21.863090 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:21.863102 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:21.865213 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
W20260926 14:45:21.865490 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:21.859732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37487 (local address 127.0.98.62:37663)
0926 14:45:21.859981 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:21.859985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:21.860007 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:45:21.860144 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:21.860146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:21.860189 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:21.860265 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:21.860656 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.861287 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.862146 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:21.862374 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:21.862458 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:21.865357 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37487: BlockingRecv error: recv got EOF from 127.0.98.1:37487 (error 108)
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":75,"threads_started":1}
Sep 26 14:45:22 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:22 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:22.044081 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:22.038566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:38913)
0926 14:45:22.038841 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:22.038854 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:22.038973 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:22.039227 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:22.039229 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:22.039404 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:22.039552 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:22.039557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.040274 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.040277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:22.040883 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:22.040892 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.041417 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.041420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:22.041503 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:22.042075 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:22.042087 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:22.043950 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
W20260926 14:45:22.044250 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:22.038629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38913 (local address 127.0.98.62:37663)
0926 14:45:22.038790 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:22.038794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:22.038910 (+ 116us) server_negotiation.cc:407] Connection header received
0926 14:45:22.039085 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:22.039088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:22.039136 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:22.039197 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:22.039641 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.040178 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.041058 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.041287 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.041373 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:22.044132 (+ 2759us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38913: BlockingRecv error: recv got EOF from 127.0.98.2:38913 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 14:45:22 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:22 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:22.252771 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:22.246916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:53147)
0926 14:45:22.247220 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:22.247236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:22.247410 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:22.247727 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:22.247730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:22.247934 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:22.248085 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:22.248093 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.249018 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.249022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:22.249596 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:22.249604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.250052 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.250056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:22.250137 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:22.250689 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:22.250701 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:22.252620 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":88,"threads_started":1}
W20260926 14:45:22.252863 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:22.247021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53147 (local address 127.0.98.62:37663)
0926 14:45:22.247190 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:22.247194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:22.247319 (+ 125us) server_negotiation.cc:407] Connection header received
0926 14:45:22.247537 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:22.247540 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:22.247597 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:22.247696 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:22.248202 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.248912 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.249707 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.249946 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.250032 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:22.252765 (+ 2733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53147: BlockingRecv error: recv got EOF from 127.0.98.3:53147 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 26 14:45:22 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:22 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:22.871606 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:22.866240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:46035)
0926 14:45:22.866602 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:22.866616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:22.866778 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:22.867022 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:22.867025 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:22.867226 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:22.867317 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:22.867323 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.868054 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.868058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:22.868606 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:22.868614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.869022 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.869026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:22.869103 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:22.869687 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:22.869725 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:22.871485 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":76,"threads_started":1}
W20260926 14:45:22.871760 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:22.866315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46035 (local address 127.0.98.62:37663)
0926 14:45:22.866561 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:22.866565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:22.866684 (+ 119us) server_negotiation.cc:407] Connection header received
0926 14:45:22.866867 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:22.866870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:22.866915 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:22.866989 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:22.867425 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.867955 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.868708 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:22.868926 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:22.869009 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:22.871629 (+ 2620us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46035: BlockingRecv error: recv got EOF from 127.0.98.1:46035 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":90,"threads_started":1}
Sep 26 14:45:23 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:23 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:23.050683 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:23.044928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:59573)
0926 14:45:23.045263 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:23.045277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:23.045381 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:23.045655 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:23.045657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:23.045838 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:23.045936 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:23.045955 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.046830 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.046840 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:23.047459 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:23.047468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.048001 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.048004 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:23.048072 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:23.048628 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:23.048640 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:23.050526 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":120,"threads_started":1}
W20260926 14:45:23.050886 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:23.044991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59573 (local address 127.0.98.62:37663)
0926 14:45:23.045141 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:23.045145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:23.045351 (+ 206us) server_negotiation.cc:407] Connection header received
0926 14:45:23.045463 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:23.045465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:23.045509 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:23.045624 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:23.046094 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.046662 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.047630 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.047873 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.047958 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:23.050740 (+ 2782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59573: BlockingRecv error: recv got EOF from 127.0.98.2:59573 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 14:45:23 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:23 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:23.259259 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:23.253660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:44885)
0926 14:45:23.254077 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:23.254090 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:23.254232 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:23.254481 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:23.254484 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:23.254680 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:23.254841 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:23.254848 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.255690 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.255694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:23.256257 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:23.256266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.256717 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.256721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:23.256802 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:23.257303 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:23.257315 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:23.259080 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":94,"threads_started":1}
W20260926 14:45:23.259342 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:23.253781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44885 (local address 127.0.98.62:37663)
0926 14:45:23.253907 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:23.253912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:23.254174 (+ 262us) server_negotiation.cc:407] Connection header received
0926 14:45:23.254332 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:23.254334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:23.254380 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:23.254448 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:23.254959 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.255589 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.256375 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.256611 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.256704 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:23.259244 (+ 2540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44885: BlockingRecv error: recv got EOF from 127.0.98.3:44885 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 14:45:23 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:23 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:23.877867 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:23.872320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:46775)
0926 14:45:23.872571 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:23.872586 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:23.872689 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:23.872928 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:23.872930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:23.873100 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:23.873194 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:23.873200 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.873940 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.873943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:23.874535 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:23.874545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.875127 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.875130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:23.875198 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:23.875741 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:23.875754 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:23.877744 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":68,"threads_started":1}
W20260926 14:45:23.877981 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:23.872477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46775 (local address 127.0.98.62:37663)
0926 14:45:23.872713 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:23.872716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:23.872747 (+ 31us) server_negotiation.cc:407] Connection header received
0926 14:45:23.872785 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:23.872787 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:23.872829 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:23.872896 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:23.873299 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.873835 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.874662 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:23.874970 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:23.875054 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:23.877861 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46775: BlockingRecv error: recv got EOF from 127.0.98.1:46775 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":90,"threads_started":1}
Sep 26 14:45:24 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:24 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:24.057951 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:24.051535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:46369)
0926 14:45:24.051847 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:24.051862 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:24.051980 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:24.052261 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:24.052263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:24.052445 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:24.052546 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:24.052552 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.053688 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.053691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:24.054364 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:24.054376 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.054974 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.054977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:24.055062 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:24.055638 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:24.055650 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:24.057789 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":92,"threads_started":1}
W20260926 14:45:24.058066 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:24.051674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46369 (local address 127.0.98.62:37663)
0926 14:45:24.051833 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:24.051837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:24.051933 (+ 96us) server_negotiation.cc:407] Connection header received
0926 14:45:24.052060 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:24.052062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:24.052106 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:24.052177 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:24.052749 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.053553 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.054503 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.054847 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.054935 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:24.057943 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46369: BlockingRecv error: recv got EOF from 127.0.98.2:46369 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 14:45:24 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:24 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:24.266960 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:24.259991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:43073)
0926 14:45:24.260273 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:24.260286 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:24.260406 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:24.260748 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:24.260751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:24.260994 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:24.261104 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:24.261111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.262028 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.262032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:24.263055 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:24.263071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.263691 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.263696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:24.263800 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:24.264375 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:24.264391 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:24.266799 (+ 2408us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
W20260926 14:45:24.267086 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:24.260074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43073 (local address 127.0.98.62:37663)
0926 14:45:24.260225 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:24.260251 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:24.260385 (+ 134us) server_negotiation.cc:407] Connection header received
0926 14:45:24.260486 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:24.260489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:24.260540 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:24.260613 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:24.261224 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.261855 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.263191 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.263547 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.263648 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:24.266982 (+ 3334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43073: BlockingRecv error: recv got EOF from 127.0.98.3:43073 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 14:45:24 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:24 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:24.884507 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:24.878727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:46403)
0926 14:45:24.879073 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:24.879090 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:24.879204 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:24.879466 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:24.879469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:24.879648 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:24.879749 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:24.879754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.880582 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.880585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:24.881201 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:24.881212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.881655 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.881658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:24.881723 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:24.882262 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:24.882275 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:24.884344 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":90,"threads_started":1}
W20260926 14:45:24.884644 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:24.878814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46403 (local address 127.0.98.62:37663)
0926 14:45:24.879104 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:24.879109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:24.879146 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:45:24.879312 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:24.879314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:24.879356 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:24.879431 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:24.879891 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.880457 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.881306 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:24.881558 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:24.881636 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:24.884505 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46403: BlockingRecv error: recv got EOF from 127.0.98.1:46403 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":71,"threads_started":1}
Sep 26 14:45:25 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:25 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:25.065037 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:25.058932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:45713)
0926 14:45:25.059255 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:25.059271 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:25.059391 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:25.059653 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:25.059656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:25.059919 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:25.060025 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:25.060031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.060880 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.060884 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:25.061628 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:25.061640 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.062159 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.062163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:25.062242 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:25.062726 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:25.062738 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:25.064879 (+ 2141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":91,"threads_started":1}
W20260926 14:45:25.065143 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:25.059082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45713 (local address 127.0.98.62:37663)
0926 14:45:25.059254 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:25.059258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:25.059345 (+ 87us) server_negotiation.cc:407] Connection header received
0926 14:45:25.059496 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:25.059499 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:25.059546 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:25.059615 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:25.060142 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.060694 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.061760 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.062030 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.062152 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:25.065043 (+ 2891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45713: BlockingRecv error: recv got EOF from 127.0.98.2:45713 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 26 14:45:25 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:25 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:25.273942 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:25.268113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:35631)
0926 14:45:25.268447 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:25.268467 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:25.268592 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:25.268853 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:25.268856 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:25.269057 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:25.269173 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:25.269181 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.270008 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.270012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:25.270648 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:25.270659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.271186 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.271190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:25.271282 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:25.271845 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:25.271862 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:25.273789 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":104,"threads_started":1}
W20260926 14:45:25.274055 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:25.268263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35631 (local address 127.0.98.62:37663)
0926 14:45:25.268432 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:25.268436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:25.268545 (+ 109us) server_negotiation.cc:407] Connection header received
0926 14:45:25.268692 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:25.268694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:25.268746 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:25.268814 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:25.269299 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.269838 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.270787 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.271040 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.271145 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:25.273955 (+ 2810us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35631: BlockingRecv error: recv got EOF from 127.0.98.3:35631 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 14:45:25 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:25 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:25.891242 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:25.885272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:33053)
0926 14:45:25.885566 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:25.885579 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:25.885672 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:25.885930 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:25.885932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:25.886122 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:25.886222 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:25.886228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.887078 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.887082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:25.887679 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:25.887690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.888271 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.888274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:25.888347 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:25.888905 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:25.888918 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:25.891076 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":80,"threads_started":1}
W20260926 14:45:25.891369 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:25.885403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33053 (local address 127.0.98.62:37663)
0926 14:45:25.885641 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:25.885644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:25.885659 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:45:25.885751 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:25.885753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:25.885795 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:25.885879 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:25.886362 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.886947 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.887834 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:25.888134 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:25.888234 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:25.891251 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33053: BlockingRecv error: recv got EOF from 127.0.98.1:33053 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":68,"threads_started":1}
Sep 26 14:45:26 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:26 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:26.072252 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:26.065995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:52949)
0926 14:45:26.066238 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:26.066252 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:26.066352 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:26.066657 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:26.066659 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:26.066873 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:26.066979 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:26.066985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.067860 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.067863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:26.068526 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:26.068540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.069077 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.069081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:26.069155 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:26.069751 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:26.069764 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:26.072074 (+ 2310us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
W20260926 14:45:26.072429 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:26.066040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52949 (local address 127.0.98.62:37663)
0926 14:45:26.066191 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:26.066195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:26.066320 (+ 125us) server_negotiation.cc:407] Connection header received
0926 14:45:26.066428 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:26.066430 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:26.066475 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:26.066565 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:26.067132 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.067714 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.068686 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.068929 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.069030 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:26.072298 (+ 3268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52949: BlockingRecv error: recv got EOF from 127.0.98.2:52949 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 14:45:26 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:26 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:26.287214 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:26.274875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:43943)
0926 14:45:26.275224 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:26.275240 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:26.275354 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:26.275636 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:26.275639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:26.275832 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:26.275929 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:26.275934 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.276867 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.276871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:26.277456 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:26.277465 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.277938 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.277942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:26.278025 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:26.284893 (+ 6868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:26.284908 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:26.286973 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":147,"threads_started":1}
W20260926 14:45:26.287264 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:26.275021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43943 (local address 127.0.98.62:37663)
0926 14:45:26.275178 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:26.275183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:26.275312 (+ 129us) server_negotiation.cc:407] Connection header received
0926 14:45:26.275450 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:26.275453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:26.275505 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:26.275589 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:26.276053 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.276741 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.277588 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.277822 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.277912 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:26.287161 (+ 9249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43943: BlockingRecv error: recv got EOF from 127.0.98.3:43943 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 14:45:26 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:26 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:26.897686 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:26.892157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:42779)
0926 14:45:26.892445 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:26.892459 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:26.892554 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:26.892803 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:26.892805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:26.892980 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:26.893081 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:26.893087 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.893885 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.893888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:26.894487 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:26.894497 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.894965 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.894968 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:26.895032 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:26.895560 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:26.895572 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:26.897538 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":79,"threads_started":1}
W20260926 14:45:26.897832 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:26.892271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42779 (local address 127.0.98.62:37663)
0926 14:45:26.892501 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:26.892504 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:26.892519 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:45:26.892642 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:26.892645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:26.892690 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:26.892771 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:26.893224 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.893782 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.894611 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:26.894875 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:26.894956 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:26.897706 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42779: BlockingRecv error: recv got EOF from 127.0.98.1:42779 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":69,"threads_started":1}
Sep 26 14:45:27 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:27 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:27.078889 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:27.073247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:58797)
0926 14:45:27.073567 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:27.073584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:27.073728 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:27.074002 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:27.074004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:27.074245 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:27.074339 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:27.074345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.075189 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.075193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:27.075758 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:27.075766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.076233 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.076236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:27.076315 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:27.076838 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:27.076850 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:27.078672 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":99,"threads_started":1}
W20260926 14:45:27.078943 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:27.073391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58797 (local address 127.0.98.62:37663)
0926 14:45:27.073577 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:27.073582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:27.073674 (+ 92us) server_negotiation.cc:407] Connection header received
0926 14:45:27.073838 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:27.073841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:27.073887 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:27.073962 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:27.074465 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.075076 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.075893 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.076119 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.076209 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:27.078845 (+ 2636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58797: BlockingRecv error: recv got EOF from 127.0.98.2:58797 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 26 14:45:27 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:27 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:27.294289 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:27.288213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:50549)
0926 14:45:27.288649 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:27.288664 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:27.288796 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:27.289065 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:27.289068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:27.289293 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:27.289390 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:27.289398 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.290189 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.290193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:27.290851 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:27.290861 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.291352 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.291356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:27.291442 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:27.292035 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:27.292048 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:27.294093 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":103,"threads_started":1}
W20260926 14:45:27.294415 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:27.288352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50549 (local address 127.0.98.62:37663)
0926 14:45:27.288576 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:27.288581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:27.288754 (+ 173us) server_negotiation.cc:407] Connection header received
0926 14:45:27.288894 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:27.288897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:27.288950 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:27.289018 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:27.289516 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.290065 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.290984 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.291227 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.291324 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:27.294306 (+ 2982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50549: BlockingRecv error: recv got EOF from 127.0.98.3:50549 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 26 14:45:27 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:27 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:27.904063 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:27.898472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:51825)
0926 14:45:27.898815 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:27.898833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:27.898941 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:27.899167 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:27.899169 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:27.899351 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:27.899448 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:27.899454 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.900283 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.900286 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:27.900865 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:27.900874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.901364 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.901367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:27.901435 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:27.901958 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:27.901971 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:27.903938 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":89,"threads_started":1}
W20260926 14:45:27.904238 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:27.898584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51825 (local address 127.0.98.62:37663)
0926 14:45:27.898846 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:27.898849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:27.898909 (+ 60us) server_negotiation.cc:407] Connection header received
0926 14:45:27.899016 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:27.899018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:27.899061 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:27.899140 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:27.899544 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.900105 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.901003 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:27.901245 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:27.901335 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:27.904089 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51825: BlockingRecv error: recv got EOF from 127.0.98.1:51825 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":54,"threads_started":1}
Sep 26 14:45:28 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:28 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:28.085493 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:28.079747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:53915)
0926 14:45:28.080054 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:28.080067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:28.080185 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:28.080495 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:28.080497 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:28.080684 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:28.080783 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:28.080789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.081652 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.081654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:28.082245 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:28.082253 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.082741 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.082744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:28.082869 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:28.083406 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:28.083418 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:28.085367 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":88,"threads_started":1}
W20260926 14:45:28.085686 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:28.079870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53915 (local address 127.0.98.62:37663)
0926 14:45:28.080042 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:28.080045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:28.080154 (+ 109us) server_negotiation.cc:407] Connection header received
0926 14:45:28.080335 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:28.080338 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:28.080386 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:28.080458 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:28.080953 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.081553 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.082385 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.082623 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.082712 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:28.085558 (+ 2846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53915: BlockingRecv error: recv got EOF from 127.0.98.2:53915 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 14:45:28 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:28 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:28.300590 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:28.295114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:36575)
0926 14:45:28.295452 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:28.295465 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:28.295563 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:28.295807 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:28.295809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:28.295989 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:28.296089 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:28.296094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.296853 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.296856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:28.297434 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:28.297442 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.297918 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.297921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:28.297994 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:28.298556 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:28.298568 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:28.300470 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":95,"threads_started":1}
W20260926 14:45:28.300762 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:28.295201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36575 (local address 127.0.98.62:37663)
0926 14:45:28.295324 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:28.295328 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:28.295557 (+ 229us) server_negotiation.cc:407] Connection header received
0926 14:45:28.295632 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:28.295634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:28.295684 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:28.295757 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:28.296185 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.296734 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.297533 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.297765 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.297866 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:28.300623 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36575: BlockingRecv error: recv got EOF from 127.0.98.3:36575 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 14:45:28 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:28 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:28.910312 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:28.904854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:46079)
0926 14:45:28.905110 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:28.905125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:28.905215 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:28.905504 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:28.905507 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:28.905687 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:28.905788 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:28.905794 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.906658 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.906661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:28.907310 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:28.907323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.907855 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.907858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:28.907921 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:28.908463 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:28.908475 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:28.910189 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
W20260926 14:45:28.910545 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:28.904947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46079 (local address 127.0.98.62:37663)
0926 14:45:28.905186 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:28.905189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:28.905203 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:45:28.905295 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:28.905297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:28.905339 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:28.905424 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:28.905944 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.906529 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.907487 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:28.907731 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:28.907821 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:28.910409 (+ 2588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46079: BlockingRecv error: recv got EOF from 127.0.98.1:46079 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":68,"threads_started":1}
Sep 26 14:45:29 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:29 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:29.091828 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:29.086297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:34551)
0926 14:45:29.086593 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:29.086605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:29.086711 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:29.087050 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:29.087053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:29.087239 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:29.087345 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:29.087351 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.088123 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.088126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:29.088711 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:29.088720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.089176 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.089179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:29.089243 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:29.089744 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:29.089756 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:29.091695 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":91,"threads_started":1}
W20260926 14:45:29.091982 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:29.086468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34551 (local address 127.0.98.62:37663)
0926 14:45:29.086597 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:29.086601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:29.086670 (+ 69us) server_negotiation.cc:407] Connection header received
0926 14:45:29.086813 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:29.086815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:29.086866 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:29.086974 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:29.087484 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.088031 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.088852 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.089079 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.089163 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:29.091859 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34551: BlockingRecv error: recv got EOF from 127.0.98.2:34551 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 14:45:29 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:29 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:29.306813 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:29.301410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:43639)
0926 14:45:29.301683 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:29.301696 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:29.301802 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:29.302040 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:29.302042 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:29.302218 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:29.302319 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:29.302324 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.303204 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.303207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:29.303775 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:29.303784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.304201 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.304203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:29.304265 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:29.304794 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:29.304806 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:29.306653 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":87,"threads_started":1}
W20260926 14:45:29.306962 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:29.301504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43639 (local address 127.0.98.62:37663)
0926 14:45:29.301690 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:29.301694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:29.301791 (+ 97us) server_negotiation.cc:407] Connection header received
0926 14:45:29.301888 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:29.301891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:29.301941 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:29.302002 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:29.302474 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.303097 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.303878 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.304107 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.304183 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:29.306839 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43639: BlockingRecv error: recv got EOF from 127.0.98.3:43639 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 26 14:45:29 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:29 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:29.916590 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:29.911151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:52815)
0926 14:45:29.911419 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:29.911433 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:29.911568 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:29.911850 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:29.911853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:29.912036 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:29.912137 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:29.912143 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.912886 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.912889 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:29.913503 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:29.913513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.913952 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.913955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:29.914048 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:29.914640 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:29.914656 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:29.916468 (+ 1812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
W20260926 14:45:29.916787 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:29.911239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52815 (local address 127.0.98.62:37663)
0926 14:45:29.911518 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:29.911522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:29.911538 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:45:29.911651 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:29.911653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:29.911696 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:29.911775 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:29.912242 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.912773 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.913603 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:29.913823 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:29.913898 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:29.916650 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52815: BlockingRecv error: recv got EOF from 127.0.98.1:52815 (error 108)
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":69,"threads_started":1}
Sep 26 14:45:30 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:30 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:30.098004 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:30.092634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:44091)
0926 14:45:30.092891 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:30.092905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:30.093029 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:30.093256 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:30.093258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:30.093434 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:30.093535 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:30.093541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.094359 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.094362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:30.095029 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:30.095041 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.095512 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.095515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:30.095582 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:30.096103 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:30.096114 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:30.097862 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
W20260926 14:45:30.098188 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:30.092737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44091 (local address 127.0.98.62:37663)
0926 14:45:30.092888 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:30.092892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:30.092969 (+ 77us) server_negotiation.cc:407] Connection header received
0926 14:45:30.093111 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:30.093113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:30.093158 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:30.093231 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:30.093672 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.094241 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.095162 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.095397 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.095487 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:30.098065 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44091: BlockingRecv error: recv got EOF from 127.0.98.2:44091 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 14:45:30 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:30 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:30.312897 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:30.307583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.3:52279)
0926 14:45:30.307830 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:30.307843 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:30.307941 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:30.308167 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:30.308169 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:30.308340 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:30.308437 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:30.308443 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.309252 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.309255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:30.309857 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:30.309866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.310336 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.310339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:30.310401 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:30.310987 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:30.310999 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:30.312766 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":77,"threads_started":1}
W20260926 14:45:30.313087 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:30.307688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52279 (local address 127.0.98.62:37663)
0926 14:45:30.307820 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:30.307824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:30.307907 (+ 83us) server_negotiation.cc:407] Connection header received
0926 14:45:30.308013 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:30.308015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:30.308062 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:30.308135 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:30.308571 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.309134 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.310003 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.310232 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.310324 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:30.312959 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52279: BlockingRecv error: recv got EOF from 127.0.98.3:52279 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 14:45:30 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:30 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:30.923043 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:30.917427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.1:33549)
0926 14:45:30.917749 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:30.917762 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:30.917861 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:30.918095 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:30.918097 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:30.918284 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:30.918381 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:30.918387 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.919287 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.919290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:30.919861 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:30.919870 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.920298 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.920301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:30.920363 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:30.920986 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:30.921000 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:30.922835 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":84,"threads_started":1}
W20260926 14:45:30.923110 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:30.917537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33549 (local address 127.0.98.62:37663)
0926 14:45:30.917755 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:30.917758 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:30.917818 (+ 60us) server_negotiation.cc:407] Connection header received
0926 14:45:30.917937 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:30.917939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:30.917982 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:30.918055 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:30.918604 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.919176 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.919977 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:30.920208 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:30.920293 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:30.922997 (+ 2704us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33549: BlockingRecv error: recv got EOF from 127.0.98.1:33549 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":66,"threads_started":1}
Sep 26 14:45:31 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 14:45:31 dist-test-slave-s7k6 krb5kdc[8526](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 14:45:31.104564 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:31.099041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37663 (local address 127.0.98.2:40111)
0926 14:45:31.099330 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:31.099343 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:31.099443 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:31.099721 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:31.099723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:31.099899 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:31.100006 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:31.100013 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.100829 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.100832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:31.101464 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:31.101475 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.101968 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.101971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:31.102040 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:31.102587 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:31.102600 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:31.104393 (+ 1793us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:37663: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":87,"threads_started":1}
W20260926 14:45:31.104646 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:31.099178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40111 (local address 127.0.98.62:37663)
0926 14:45:31.099334 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:31.099337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:31.099438 (+ 101us) server_negotiation.cc:407] Connection header received
0926 14:45:31.099523 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:31.099526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:31.099573 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:31.099684 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:31.100122 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.100736 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.101586 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.101851 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.101935 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:31.104547 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40111: BlockingRecv error: recv got EOF from 127.0.98.2:40111 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 14:45:31.193078 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 9109
I20260926 14:45:31.197636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 9261
I20260926 14:45:31.202461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 9393
I20260926 14:45:31.207042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 9039
2026-09-26T14:45:31Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17911 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T14:45:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:31Z Disabled control of system clock
I20260926 14:45:31.234690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43105
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43105
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 14:45:31.341924 9589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:31.342208 9589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:31.342244 9589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:31.346236 9589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:31.346298 9589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:31.346312 9589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:31.346326 9589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:31.346340 9589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:45:31.351341 9589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43105
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:31.352353 9589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:31.353681 9589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:31.361639 9597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:31.361680 9594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:31.361680 9595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:31.361990 9589 server_base.cc:1034] running on GCE node
I20260926 14:45:31.362535 9589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:31.363617 9589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:31.364786 9589 hybrid_clock.cc:648] HybridClock initialized: now 1790433931364769 us; error 29 us; skew 500 ppm
I20260926 14:45:31.371471 9589 webserver.cc:460] Webserver started at http://127.0.98.62:34501/ using document root <none> and password file <none>
I20260926 14:45:31.372098 9589 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:31.372171 9589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:31.372375 9589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:31.374418 9589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "6191df66cbbf470796ebc43e8dde56c9"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "8338b605e02bbf6ca0b67871668fefb7"
server_key_iv: "4edfa50484bd7eba8e2b02e01f0cd811"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.374965 9589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "6191df66cbbf470796ebc43e8dde56c9"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "8338b605e02bbf6ca0b67871668fefb7"
server_key_iv: "4edfa50484bd7eba8e2b02e01f0cd811"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.378540 9589 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:45:31.381109 9603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:31.382467 9589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:45:31.382584 9589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "6191df66cbbf470796ebc43e8dde56c9"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "8338b605e02bbf6ca0b67871668fefb7"
server_key_iv: "4edfa50484bd7eba8e2b02e01f0cd811"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.382683 9589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:31.394292 9589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:31.394981 9589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:31.395150 9589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:31.401701 9589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43105
I20260926 14:45:31.401736 9655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43105 every 8 connection(s)
I20260926 14:45:31.402588 9589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:31.405558 9656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:31.410008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 9589
I20260926 14:45:31.410115 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:45:31.410363 392 external_mini_cluster.cc:1428] Setting key a9129c2fca0195468a9c525b4ca5c59d
I20260926 14:45:31.411116 9656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9: Bootstrap starting.
I20260926 14:45:31.413666 9656 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:31.414557 9656 log.cc:824] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:31.416651 9656 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9: No bootstrap required, opened a new log
I20260926 14:45:31.420885 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:31.411823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.0.98.62:43105)
0926 14:45:31.412534 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:31.412552 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:31.412594 (+ 42us) server_negotiation.cc:407] Connection header received
0926 14:45:31.413411 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:31.413435 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:31.413979 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:31.414338 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:31.414829 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.416155 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.417116 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.417787 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.417943 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:31.418738 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:31.418744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:31.418783 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:31.419269 (+ 486us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:31.419337 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:31.419363 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:31.419459 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:31.419619 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 14:45:31.419912 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":116,"threads_started":1}
I20260926 14:45:31.420998 9656 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6191df66cbbf470796ebc43e8dde56c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } }
I20260926 14:45:31.421253 9656 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:31.421308 9656 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6191df66cbbf470796ebc43e8dde56c9, State: Initialized, Role: FOLLOWER
I20260926 14:45:31.421752 9656 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6191df66cbbf470796ebc43e8dde56c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } }
I20260926 14:45:31.421875 9656 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:31.421929 9656 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:31.422019 9656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:31.422890 9656 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6191df66cbbf470796ebc43e8dde56c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } }
I20260926 14:45:31.423218 9656 leader_election.cc:302] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6191df66cbbf470796ebc43e8dde56c9; no voters:
I20260926 14:45:31.423499 9656 leader_election.cc:288] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:45:31.423645 9661 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:31.423843 9661 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [term 1 LEADER]: Becoming Leader. State: Replica: 6191df66cbbf470796ebc43e8dde56c9, State: Running, Role: LEADER
I20260926 14:45:31.424150 9661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6191df66cbbf470796ebc43e8dde56c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } }
I20260926 14:45:31.424679 9656 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:31.425787 9662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6191df66cbbf470796ebc43e8dde56c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6191df66cbbf470796ebc43e8dde56c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } }
I20260926 14:45:31.425953 9662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:31.425787 9663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6191df66cbbf470796ebc43e8dde56c9. Latest consensus state: current_term: 1 leader_uuid: "6191df66cbbf470796ebc43e8dde56c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6191df66cbbf470796ebc43e8dde56c9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43105 } } }
I20260926 14:45:31.426034 9663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9 [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:31.426452 9670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:31.429504 9670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:31.435483 9670 catalog_manager.cc:1347] Generated new cluster ID: 26027a4ff4f74b829a6bce11159b1ce4
I20260926 14:45:31.435559 9670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:31.450150 9670 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:45:31.451294 9670 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:31.461934 9670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6191df66cbbf470796ebc43e8dde56c9: Generated new TSK 0
I20260926 14:45:31.462667 9670 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:45:31.465672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--builtin_ntp_servers=127.0.98.20:45213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 14:45:31.578487 9680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:31.578800 9680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:31.578840 9680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:31.583091 9680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:31.583145 9680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:31.583209 9680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:45:31.588230 9680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:31.589309 9680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:31.590727 9680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:31.600303 9685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:31.600369 9686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:31.600448 9688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:31.600502 9680 server_base.cc:1034] running on GCE node
I20260926 14:45:31.601068 9680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:31.601720 9680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:31.602857 9680 hybrid_clock.cc:648] HybridClock initialized: now 1790433931602827 us; error 50 us; skew 500 ppm
I20260926 14:45:31.604861 9680 webserver.cc:460] Webserver started at http://127.0.98.1:33441/ using document root <none> and password file <none>
I20260926 14:45:31.605441 9680 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:31.605508 9680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:31.605712 9680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:31.607897 9680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "9070ff7a31b94c66bf5ba0dbafc0d09c"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "1ce5151f05ffe67828c9e479a259bb34"
server_key_iv: "7c502ab64b348b797b73b2fae2f0279f"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.608443 9680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9070ff7a31b94c66bf5ba0dbafc0d09c"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "1ce5151f05ffe67828c9e479a259bb34"
server_key_iv: "7c502ab64b348b797b73b2fae2f0279f"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.612397 9680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:45:31.615175 9694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:31.616478 9680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:45:31.616585 9680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "9070ff7a31b94c66bf5ba0dbafc0d09c"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "1ce5151f05ffe67828c9e479a259bb34"
server_key_iv: "7c502ab64b348b797b73b2fae2f0279f"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.616711 9680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:31.627265 9680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:31.627915 9680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:31.628057 9680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:31.628561 9680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:31.629436 9680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:31.629482 9680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:31.629519 9680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:31.629535 9680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:31.637518 9680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33181
I20260926 14:45:31.637595 9807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33181 every 8 connection(s)
I20260926 14:45:31.638415 9680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:45:31.642023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 9680
I20260926 14:45:31.642140 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:45:31.642484 392 external_mini_cluster.cc:1428] Setting key 36cf3f352fd5cc5202e3ce538873911e
I20260926 14:45:31.644512 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--builtin_ntp_servers=127.0.98.20:45213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 14:45:31.648563 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:31.640248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59935 (local address 127.0.98.62:43105)
0926 14:45:31.640396 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:31.640400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:31.641431 (+ 1031us) server_negotiation.cc:407] Connection header received
0926 14:45:31.643138 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:31.643142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:31.643196 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:31.643266 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:31.644312 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.644994 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.646391 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.646736 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.646870 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:31.647978 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:31.647982 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:31.647985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:31.648000 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:31.648025 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:31.648036 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:31.648127 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:31.648331 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 14:45:31.648404 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 14:45:31.649441 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:31.640610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43105 (local address 127.0.98.1:59935)
0926 14:45:31.641282 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:31.641322 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:31.642907 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:31.643446 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:31.643453 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:31.643537 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:31.644158 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:31.644178 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.645129 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.645133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:31.646234 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:31.646246 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.646918 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.646922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:31.647105 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:31.647703 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:45:31.647705 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:31.647747 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:45:31.647815 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:31.648167 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:31.648176 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:31.648181 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:45:31.648311 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 14:45:31.648568 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260926 14:45:31.650693 9808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43105
I20260926 14:45:31.650956 9808 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:31.651507 9808 heartbeater.cc:511] Master 127.0.98.62:43105 requested a full tablet report, sending...
I20260926 14:45:31.653031 9620 ts_manager.cc:194] Registered new tserver with Master: 9070ff7a31b94c66bf5ba0dbafc0d09c (127.0.98.1:33181)
I20260926 14:45:31.654172 9620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:59935
W20260926 14:45:31.756503 9811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:31.756790 9811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:31.756845 9811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:31.760953 9811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:31.761009 9811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:31.761085 9811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:45:31.765973 9811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9811
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:31.767097 9811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:31.768477 9811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:31.778017 9817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:31.778019 9819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:31.778069 9811 server_base.cc:1034] running on GCE node
W20260926 14:45:31.778017 9816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:31.778642 9811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:31.779343 9811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:31.780512 9811 hybrid_clock.cc:648] HybridClock initialized: now 1790433931780493 us; error 30 us; skew 500 ppm
I20260926 14:45:31.782299 9811 webserver.cc:460] Webserver started at http://127.0.98.2:34307/ using document root <none> and password file <none>
I20260926 14:45:31.782932 9811 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:31.783004 9811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:31.783205 9811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:31.785394 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "900511c206854d34bad592ff33a2a3fa"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "d3acaa0a891677041d0a6fe212a4e30c"
server_key_iv: "0c1726642d32b730377cc8168020a601"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.785941 9811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "900511c206854d34bad592ff33a2a3fa"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "d3acaa0a891677041d0a6fe212a4e30c"
server_key_iv: "0c1726642d32b730377cc8168020a601"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.789783 9811 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 14:45:31.792384 9825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:31.793726 9811 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:45:31.793838 9811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "900511c206854d34bad592ff33a2a3fa"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "d3acaa0a891677041d0a6fe212a4e30c"
server_key_iv: "0c1726642d32b730377cc8168020a601"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.793948 9811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:31.816342 9811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:31.817087 9811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:31.817283 9811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:31.817909 9811 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:31.819033 9811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:31.819085 9811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:31.819142 9811 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:31.819167 9811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:31.827641 9811 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46363
I20260926 14:45:31.827673 9938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46363 every 8 connection(s)
I20260926 14:45:31.828521 9811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:45:31.831171 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 9811
I20260926 14:45:31.831291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:45:31.831593 392 external_mini_cluster.cc:1428] Setting key f9868020a33c5d2e372045c8388ec926
I20260926 14:45:31.833554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--builtin_ntp_servers=127.0.98.20:45213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 14:45:31.838044 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:31.830413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43423 (local address 127.0.98.62:43105)
0926 14:45:31.830544 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:31.830548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:31.831862 (+ 1314us) server_negotiation.cc:407] Connection header received
0926 14:45:31.833471 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:31.833474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:31.833523 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:31.833591 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:31.834497 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.835079 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.836193 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.836415 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.836501 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:31.837599 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:31.837602 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:31.837604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:31.837615 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:31.837634 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:31.837641 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:31.837698 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:31.837843 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 14:45:31.837903 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:45:31.838948 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:31.830923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43105 (local address 127.0.98.2:43423)
0926 14:45:31.831713 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:31.831756 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:31.833274 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:31.833766 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:31.833774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:31.833861 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:31.834351 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:31.834363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.835196 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.835200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:31.836073 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:31.836083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:31.836552 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:31.836556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:31.836717 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:31.837346 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:45:31.837348 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:31.837391 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:45:31.837456 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:31.837711 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:31.837719 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:31.837725 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:45:31.837833 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 14:45:31.838125 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":134,"threads_started":1}
I20260926 14:45:31.840240 9939 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43105
I20260926 14:45:31.840526 9939 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:31.841166 9939 heartbeater.cc:511] Master 127.0.98.62:43105 requested a full tablet report, sending...
I20260926 14:45:31.842496 9620 ts_manager.cc:194] Registered new tserver with Master: 900511c206854d34bad592ff33a2a3fa (127.0.98.2:46363)
I20260926 14:45:31.843132 9620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43423
W20260926 14:45:31.948592 9942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:31.948889 9942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:31.948943 9942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:31.953106 9942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:31.953173 9942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:31.953250 9942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:45:31.958205 9942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:31.959527 9942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:31.961058 9942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:31.971103 9948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:31.971138 9950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:31.971138 9947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:31.971329 9942 server_base.cc:1034] running on GCE node
I20260926 14:45:31.971895 9942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:31.972576 9942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:31.973762 9942 hybrid_clock.cc:648] HybridClock initialized: now 1790433931973753 us; error 34 us; skew 500 ppm
I20260926 14:45:31.975924 9942 webserver.cc:460] Webserver started at http://127.0.98.3:34789/ using document root <none> and password file <none>
I20260926 14:45:31.976519 9942 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:31.976583 9942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:31.976789 9942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:31.978971 9942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "cc19cc1d6e8946f0ba237864df971e6b"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "f3fe03e0c50230343d607a06b672688c"
server_key_iv: "c1c18473e78ac956bb5e90f36b1fc69d"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.979529 9942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cc19cc1d6e8946f0ba237864df971e6b"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "f3fe03e0c50230343d607a06b672688c"
server_key_iv: "c1c18473e78ac956bb5e90f36b1fc69d"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.983412 9942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:45:31.986009 9956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:31.987383 9942 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:45:31.987494 9942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "cc19cc1d6e8946f0ba237864df971e6b"
format_stamp: "Formatted at 2026-09-26 14:45:31 on dist-test-slave-s7k6"
server_key: "f3fe03e0c50230343d607a06b672688c"
server_key_iv: "c1c18473e78ac956bb5e90f36b1fc69d"
server_key_version: "encryptionkey@0"
I20260926 14:45:31.987604 9942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:32.006536 9942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:32.007274 9942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:32.007448 9942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:32.008035 9942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:32.008994 9942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:32.009043 9942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.009096 9942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:32.009125 9942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.017503 9942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35905
I20260926 14:45:32.017524 10069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35905 every 8 connection(s)
I20260926 14:45:32.018461 9942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:45:32.019533 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 9942
I20260926 14:45:32.019622 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:45:32.019874 392 external_mini_cluster.cc:1428] Setting key d9d429caef281a1e174a502c9c5842a6
I20260926 14:45:32.028321 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:32.020447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45349 (local address 127.0.98.62:43105)
0926 14:45:32.020596 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:32.020600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:32.021957 (+ 1357us) server_negotiation.cc:407] Connection header received
0926 14:45:32.023376 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:32.023379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:32.023427 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:32.023494 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:32.024361 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:32.024897 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:32.026231 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:32.026465 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:32.026569 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:32.027798 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:32.027801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:32.027803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:32.027815 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:32.027836 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:32.027844 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:32.027921 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:32.028143 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 14:45:32.028200 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:45:32.029183 10072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:32.021031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43105 (local address 127.0.98.3:45349)
0926 14:45:32.021821 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:32.021874 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:32.023184 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:32.023638 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:32.023645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:32.023735 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:32.024235 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:32.024247 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:32.025059 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:32.025063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:32.026110 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:32.026120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:32.026604 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:32.026609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:32.026862 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:32.027481 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:45:32.027482 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:32.027525 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:45:32.027602 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:32.027961 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:32.027969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:32.027974 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:45:32.028091 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 14:45:32.028361 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":127,"threads_started":1}
I20260926 14:45:32.030349 10070 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43105
I20260926 14:45:32.030561 10070 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:32.031136 10070 heartbeater.cc:511] Master 127.0.98.62:43105 requested a full tablet report, sending...
I20260926 14:45:32.032147 9620 ts_manager.cc:194] Registered new tserver with Master: cc19cc1d6e8946f0ba237864df971e6b (127.0.98.3:35905)
I20260926 14:45:32.032796 9620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45349
I20260926 14:45:32.033684 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:45:32.036628 9659 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:32.035771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51696 (local address 127.0.98.62:43105)
0926 14:45:32.035886 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:32.035890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:32.036061 (+ 171us) server_negotiation.cc:407] Connection header received
0926 14:45:32.036173 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:32.036176 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:32.036218 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:32.036284 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:32.036522 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51696: BlockingRecv error: recv got EOF from 127.0.0.1:51696 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:45:32.036936 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 9680
I20260926 14:45:32.042706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 9811
I20260926 14:45:32.047466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 9942
I20260926 14:45:32.052011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 9589
2026-09-26T14:45:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (843 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T14:45:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:32Z Disabled control of system clock
I20260926 14:45:32.078574 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43373
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:45:32.192629 10081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:32.192938 10081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:32.192977 10081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:32.197286 10081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:32.197366 10081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:32.197381 10081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:32.197396 10081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:32.197410 10081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:45:32.202553 10081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:32.203682 10081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:32.205096 10081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:32.214144 10087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:32.214164 10089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:32.214164 10086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:32.214558 10081 server_base.cc:1034] running on GCE node
I20260926 14:45:32.215247 10081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:32.216492 10081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:32.217671 10081 hybrid_clock.cc:648] HybridClock initialized: now 1790433932217658 us; error 38 us; skew 500 ppm
I20260926 14:45:32.219820 10081 webserver.cc:460] Webserver started at http://127.0.98.62:45555/ using document root <none> and password file <none>
I20260926 14:45:32.220463 10081 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:32.220538 10081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:32.220741 10081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:32.222985 10081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "ca6dc12a4e7640bb85c6db8a1f032c9d"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "462a8ec035a4322ef53e1ab8621db770"
server_key_iv: "0ea708f8564e2354f5d80b228829ff28"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.223572 10081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "ca6dc12a4e7640bb85c6db8a1f032c9d"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "462a8ec035a4322ef53e1ab8621db770"
server_key_iv: "0ea708f8564e2354f5d80b228829ff28"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.227386 10081 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:45:32.229909 10095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.231251 10081 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:45:32.231390 10081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "ca6dc12a4e7640bb85c6db8a1f032c9d"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "462a8ec035a4322ef53e1ab8621db770"
server_key_iv: "0ea708f8564e2354f5d80b228829ff28"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.231498 10081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:32.236202 10081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:32.236809 10081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:32.236979 10081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:32.243572 10081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43373
I20260926 14:45:32.243604 10147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43373 every 8 connection(s)
I20260926 14:45:32.244567 10081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:32.247910 10148 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:32.253943 10148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d: Bootstrap starting.
I20260926 14:45:32.254195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 10081
I20260926 14:45:32.254334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:45:32.254628 392 external_mini_cluster.cc:1428] Setting key 6c00a4ea1f8e1804df14309248379d5a
I20260926 14:45:32.256872 10148 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:32.257665 10148 log.cc:824] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:32.260064 10148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d: No bootstrap required, opened a new log
I20260926 14:45:32.260773 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:32.256195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33242 (local address 127.0.98.62:43373)
0926 14:45:32.256863 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:32.256879 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:32.256918 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:45:32.257629 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:32.257648 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:32.258082 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:32.258482 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:32.259296 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:32.259301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:32.259313 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:32.259329 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:32.259388 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:32.259406 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:32.259494 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:32.259681 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 14:45:32.260004 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":127,"threads_started":1}
I20260926 14:45:32.265861 10148 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6dc12a4e7640bb85c6db8a1f032c9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } }
I20260926 14:45:32.266243 10148 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:32.266341 10148 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca6dc12a4e7640bb85c6db8a1f032c9d, State: Initialized, Role: FOLLOWER
I20260926 14:45:32.266889 10148 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6dc12a4e7640bb85c6db8a1f032c9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } }
I20260926 14:45:32.267050 10148 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:32.267141 10148 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:32.267251 10148 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:32.268582 10148 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6dc12a4e7640bb85c6db8a1f032c9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } }
I20260926 14:45:32.268998 10148 leader_election.cc:302] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca6dc12a4e7640bb85c6db8a1f032c9d; no voters:
I20260926 14:45:32.269405 10148 leader_election.cc:288] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:45:32.269526 10153 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:32.269753 10153 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [term 1 LEADER]: Becoming Leader. State: Replica: ca6dc12a4e7640bb85c6db8a1f032c9d, State: Running, Role: LEADER
I20260926 14:45:32.270084 10153 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6dc12a4e7640bb85c6db8a1f032c9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } }
I20260926 14:45:32.270807 10148 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:32.271394 10155 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca6dc12a4e7640bb85c6db8a1f032c9d. Latest consensus state: current_term: 1 leader_uuid: "ca6dc12a4e7640bb85c6db8a1f032c9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6dc12a4e7640bb85c6db8a1f032c9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } } }
I20260926 14:45:32.271526 10155 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:32.271912 10154 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca6dc12a4e7640bb85c6db8a1f032c9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca6dc12a4e7640bb85c6db8a1f032c9d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43373 } } }
I20260926 14:45:32.272009 10154 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d [sys.catalog]: This master's current role is: LEADER
W20260926 14:45:32.275734 10168 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:45:32.275813 10168 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:45:32.275893 10162 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:32.276599 10162 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:32.282377 10162 catalog_manager.cc:1347] Generated new cluster ID: 008e02b388354e598d226c84566a8fa0
I20260926 14:45:32.282450 10162 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:32.291640 10162 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:45:32.292032 10162 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:32.300176 10162 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca6dc12a4e7640bb85c6db8a1f032c9d: Generated new TSK 0
I20260926 14:45:32.301091 10162 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:45:32.303789 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43373
--builtin_ntp_servers=127.0.98.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:45:32.416373 10172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:32.416663 10172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:32.416723 10172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:32.420897 10172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:32.420961 10172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:32.421046 10172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:45:32.426072 10172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:32.427265 10172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:32.428725 10172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:32.438534 10177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:32.438563 10178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:32.438655 10172 server_base.cc:1034] running on GCE node
W20260926 14:45:32.438563 10180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:32.439355 10172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:32.440085 10172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:32.441267 10172 hybrid_clock.cc:648] HybridClock initialized: now 1790433932441243 us; error 36 us; skew 500 ppm
I20260926 14:45:32.443338 10172 webserver.cc:460] Webserver started at http://127.0.98.1:34363/ using document root <none> and password file <none>
I20260926 14:45:32.443922 10172 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:32.443986 10172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:32.444157 10172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:32.446297 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "adf91b8cc62743f9a9471b28031f35ed"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "14d23f7b4ac5f7a36ae369dfd842dc0b"
server_key_iv: "ce32495d9d9af340e33f21e9ad79f5d2"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.446903 10172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "adf91b8cc62743f9a9471b28031f35ed"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "14d23f7b4ac5f7a36ae369dfd842dc0b"
server_key_iv: "ce32495d9d9af340e33f21e9ad79f5d2"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.450963 10172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:45:32.453826 10186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.455291 10172 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:32.455421 10172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "adf91b8cc62743f9a9471b28031f35ed"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "14d23f7b4ac5f7a36ae369dfd842dc0b"
server_key_iv: "ce32495d9d9af340e33f21e9ad79f5d2"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.455554 10172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:32.460325 10172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:32.460949 10172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:32.461109 10172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:32.461668 10172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:32.462576 10172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:32.462622 10172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.462662 10172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:32.462690 10172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.471486 10172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33987
I20260926 14:45:32.471549 10299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33987 every 8 connection(s)
I20260926 14:45:32.472517 10172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:45:32.478936 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:32.474465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55195 (local address 127.0.98.62:43373)
0926 14:45:32.474606 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:32.474610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:32.475839 (+ 1229us) server_negotiation.cc:407] Connection header received
0926 14:45:32.477157 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:32.477160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:32.477208 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:32.477293 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:32.478464 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:32.478467 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:32.478471 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:32.478483 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:32.478502 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:32.478509 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:32.478589 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:32.478727 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 14:45:32.478797 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:45:32.479807 10302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:32.474897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43373 (local address 127.0.98.1:55195)
0926 14:45:32.475605 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:32.475647 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:32.476926 (+ 1279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:32.477483 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:32.477491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:32.477587 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:32.478173 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:45:32.478175 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:32.478219 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:45:32.478297 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:32.478583 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:32.478591 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:32.478597 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:45:32.478795 (+ 198us) client_negotiation.cc:240] Negotiation successful
0926 14:45:32.479054 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":116,"threads_started":1}
I20260926 14:45:32.480436 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 10172
I20260926 14:45:32.480544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:45:32.480856 392 external_mini_cluster.cc:1428] Setting key 3ef8155160efdd8940c943f5f268f621
I20260926 14:45:32.481081 10300 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43373
I20260926 14:45:32.481323 10300 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:32.481750 10300 heartbeater.cc:511] Master 127.0.98.62:43373 requested a full tablet report, sending...
I20260926 14:45:32.483007 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43373
--builtin_ntp_servers=127.0.98.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 14:45:32.483565 10112 ts_manager.cc:194] Registered new tserver with Master: adf91b8cc62743f9a9471b28031f35ed (127.0.98.1:33987)
W20260926 14:45:32.595252 10303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:32.595531 10303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:32.595570 10303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:32.599725 10303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:32.599800 10303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:32.599885 10303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:45:32.605249 10303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:32.606365 10303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:32.607882 10303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:32.618391 10311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:32.618388 10309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:32.618388 10308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:32.618654 10303 server_base.cc:1034] running on GCE node
I20260926 14:45:32.619167 10303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:32.619922 10303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:32.621099 10303 hybrid_clock.cc:648] HybridClock initialized: now 1790433932621079 us; error 38 us; skew 500 ppm
I20260926 14:45:32.623172 10303 webserver.cc:460] Webserver started at http://127.0.98.2:41599/ using document root <none> and password file <none>
I20260926 14:45:32.623754 10303 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:32.623826 10303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:32.624019 10303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:32.626188 10303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "da7f7784cb584059b6f88df14edadb69"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "99b1d889f1c3f50ddaaaafedac30dfed"
server_key_iv: "fc4a4e9b14b97bfb73c911f82798d6bf"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.626746 10303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "da7f7784cb584059b6f88df14edadb69"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "99b1d889f1c3f50ddaaaafedac30dfed"
server_key_iv: "fc4a4e9b14b97bfb73c911f82798d6bf"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.630470 10303 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:45:32.633100 10317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.634545 10303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:45:32.634665 10303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "da7f7784cb584059b6f88df14edadb69"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "99b1d889f1c3f50ddaaaafedac30dfed"
server_key_iv: "fc4a4e9b14b97bfb73c911f82798d6bf"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.634799 10303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:32.639495 10303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:32.640118 10303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:32.640295 10303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:32.640829 10303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:32.641729 10303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:32.641777 10303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.641831 10303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:32.641860 10303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.650477 10303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35131
I20260926 14:45:32.650498 10430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35131 every 8 connection(s)
I20260926 14:45:32.651489 10303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:45:32.657740 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:32.653456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43507 (local address 127.0.98.62:43373)
0926 14:45:32.653599 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:32.653603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:32.654727 (+ 1124us) server_negotiation.cc:407] Connection header received
0926 14:45:32.656031 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:32.656034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:32.656072 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:32.656154 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:32.657361 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:32.657364 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:32.657366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:32.657378 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:32.657398 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:32.657406 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:32.657486 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:32.657590 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 14:45:32.657628 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:45:32.658711 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:32.653873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43373 (local address 127.0.98.2:43507)
0926 14:45:32.654568 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:32.654609 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:32.655824 (+ 1215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:32.656364 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:32.656372 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:32.656475 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:32.657062 (+ 587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:45:32.657064 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:32.657107 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:45:32.657181 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:32.657480 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:32.657485 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:32.657489 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:45:32.657623 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 14:45:32.657901 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260926 14:45:32.659776 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 10303
I20260926 14:45:32.659883 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:45:32.659914 10431 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43373
I20260926 14:45:32.660141 10431 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:32.660173 392 external_mini_cluster.cc:1428] Setting key b39bf2a3dbe9df27f08085c7861af5c7
I20260926 14:45:32.660562 10431 heartbeater.cc:511] Master 127.0.98.62:43373 requested a full tablet report, sending...
I20260926 14:45:32.661593 10112 ts_manager.cc:194] Registered new tserver with Master: da7f7784cb584059b6f88df14edadb69 (127.0.98.2:35131)
I20260926 14:45:32.662103 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43373
--builtin_ntp_servers=127.0.98.20:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 14:45:32.774273 10434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:32.774569 10434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:32.774627 10434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:32.778777 10434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:32.778869 10434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:32.778956 10434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:45:32.784037 10434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:32.785187 10434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:32.786590 10434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:32.796353 10439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:32.796377 10442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:32.796291 10440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:32.796895 10434 server_base.cc:1034] running on GCE node
I20260926 14:45:32.797369 10434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:32.798049 10434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:32.799233 10434 hybrid_clock.cc:648] HybridClock initialized: now 1790433932799208 us; error 40 us; skew 500 ppm
I20260926 14:45:32.801311 10434 webserver.cc:460] Webserver started at http://127.0.98.3:34867/ using document root <none> and password file <none>
I20260926 14:45:32.801945 10434 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:32.802009 10434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:32.802206 10434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:32.804400 10434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "cc08274fbb374403a833ef29654a39e9"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "38e69a8208970298669648745f8a4e8d"
server_key_iv: "36c25c2bf319aed667ce8c8ae6595597"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.805001 10434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cc08274fbb374403a833ef29654a39e9"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "38e69a8208970298669648745f8a4e8d"
server_key_iv: "36c25c2bf319aed667ce8c8ae6595597"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.809156 10434 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:45:32.811883 10448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.813310 10434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:32.813447 10434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "cc08274fbb374403a833ef29654a39e9"
format_stamp: "Formatted at 2026-09-26 14:45:32 on dist-test-slave-s7k6"
server_key: "38e69a8208970298669648745f8a4e8d"
server_key_iv: "36c25c2bf319aed667ce8c8ae6595597"
server_key_version: "encryptionkey@0"
I20260926 14:45:32.813576 10434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:32.818522 10434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:32.819195 10434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:32.819362 10434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:32.819947 10434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:32.820921 10434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:32.820971 10434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.821012 10434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:32.821043 10434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:32.829803 10434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41153
I20260926 14:45:32.829869 10561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41153 every 8 connection(s)
I20260926 14:45:32.830729 10434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:45:32.837143 10151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:32.832678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44389 (local address 127.0.98.62:43373)
0926 14:45:32.832815 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:32.832819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:32.833960 (+ 1141us) server_negotiation.cc:407] Connection header received
0926 14:45:32.835449 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:32.835452 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:32.835493 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:32.835576 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:32.836701 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:32.836704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:32.836706 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:45:32.836718 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:32.836793 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:45:32.836803 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:32.836892 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:32.836994 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 14:45:32.837033 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:45:32.838071 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:32.833098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43373 (local address 127.0.98.3:44389)
0926 14:45:32.833787 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:32.833832 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:32.835183 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:32.835741 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:32.835749 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:32.835852 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:32.836410 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:45:32.836412 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:32.836453 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:45:32.836528 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:32.836880 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:32.836885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:32.836889 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:45:32.837016 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 14:45:32.837229 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":120,"threads_started":1}
I20260926 14:45:32.838145 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 10434
I20260926 14:45:32.838240 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:45:32.838515 392 external_mini_cluster.cc:1428] Setting key 12ccb0a822bd28b24cbc625e75a064a7
I20260926 14:45:32.839329 10562 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43373
I20260926 14:45:32.839587 10562 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:32.840099 10562 heartbeater.cc:511] Master 127.0.98.62:43373 requested a full tablet report, sending...
I20260926 14:45:32.841091 10112 ts_manager.cc:194] Registered new tserver with Master: cc08274fbb374403a833ef29654a39e9 (127.0.98.3:41153)
I20260926 14:45:32.852722 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 14:45:32.855602 10151 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:45:32.854639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.0.98.62:43373)
0926 14:45:32.854785 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:32.854789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:32.854992 (+ 203us) server_negotiation.cc:407] Connection header received
0926 14:45:32.855096 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:32.855099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:32.855135 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:32.855206 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:32.855464 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33250: BlockingRecv error: recv got EOF from 127.0.0.1:33250 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:45:32.855814 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 10172
I20260926 14:45:32.860337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 10303
I20260926 14:45:32.864790 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 10434
I20260926 14:45:32.869099 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 10081
2026-09-26T14:45:32Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (817 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:45:32 dist-test-slave-s7k6 krb5kdc[10572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:45:32 dist-test-slave-s7k6 krb5kdc[10572](info): set up 2 sockets
Sep 26 14:45:32 dist-test-slave-s7k6 krb5kdc[10572](info): commencing operation
krb5kdc: starting...
W20260926 14:45:33.827533 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:45:33 dist-test-slave-s7k6 krb5kdc[10572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:45:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:45:33.945719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39383
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:34.056725 10585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:34.057020 10585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:34.057068 10585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:34.061213 10585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:45:34.061282 10585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:34.061301 10585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:34.061316 10585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:45:34.061331 10585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:45:34.066946 10585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:34.067989 10585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:34.068966 10585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:34.077210 10593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:34.077230 10591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:34.077230 10590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:34.077544 10585 server_base.cc:1034] running on GCE node
I20260926 14:45:34.078176 10585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:34.079372 10585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:34.080570 10585 hybrid_clock.cc:648] HybridClock initialized: now 1790433934080557 us; error 40 us; skew 500 ppm
Sep 26 14:45:34 dist-test-slave-s7k6 krb5kdc[10572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:34.083457 10585 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:45:34.084367 10585 webserver.cc:460] Webserver started at http://127.0.98.62:44177/ using document root <none> and password file <none>
I20260926 14:45:34.084935 10585 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:34.085006 10585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:34.085227 10585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:34.087373 10585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "1733e356025d45e6a53b9234e2acc37b"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "c368562e43c0751230ff6bf510a918ae"
server_key_iv: "964add4fdfe3fedcc7d4ddb28ff36997"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.087939 10585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "1733e356025d45e6a53b9234e2acc37b"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "c368562e43c0751230ff6bf510a918ae"
server_key_iv: "964add4fdfe3fedcc7d4ddb28ff36997"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.091923 10585 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:45:34.094687 10600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:34.096117 10585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:45:34.096247 10585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "1733e356025d45e6a53b9234e2acc37b"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "c368562e43c0751230ff6bf510a918ae"
server_key_iv: "964add4fdfe3fedcc7d4ddb28ff36997"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.096362 10585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:34.106353 10585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:34.109583 10585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:34.109768 10585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:34.116715 10585 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39383
I20260926 14:45:34.116977 10652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39383 every 8 connection(s)
I20260926 14:45:34.117725 10585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:45:34.120630 10653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:34.121587 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 10585
I20260926 14:45:34.121690 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:45:34.121942 392 external_mini_cluster.cc:1428] Setting key e9427c0469ea5f381ad541df3a833284
I20260926 14:45:34.127581 10653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b: Bootstrap starting.
I20260926 14:45:34.130128 10653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:34.130872 10653 log.cc:824] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:45:34 dist-test-slave-s7k6 krb5kdc[10572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433933, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:45:34.133023 10653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b: No bootstrap required, opened a new log
I20260926 14:45:34.136946 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.123454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56466 (local address 127.0.98.62:39383)
0926 14:45:34.123886 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:34.123905 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:34.123954 (+ 49us) server_negotiation.cc:407] Connection header received
0926 14:45:34.124794 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:34.124819 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:34.125293 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:34.125681 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:34.126383 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.127831 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.128738 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.129415 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.129574 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.132187 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:34.132193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:34.132208 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:34.132235 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:34.133978 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:34.134521 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:34.134527 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:34.134531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:34.134645 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:34.134964 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:34.134967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:34.134968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:34.135395 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:34.135539 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:34.135858 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 14:45:34.136191 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":102,"threads_started":1}
I20260926 14:45:34.136945 10653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1733e356025d45e6a53b9234e2acc37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260926 14:45:34.137248 10653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:34.137316 10653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1733e356025d45e6a53b9234e2acc37b, State: Initialized, Role: FOLLOWER
I20260926 14:45:34.137742 10653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1733e356025d45e6a53b9234e2acc37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260926 14:45:34.137862 10653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:45:34.137923 10653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:45:34.138015 10653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:34.138947 10653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1733e356025d45e6a53b9234e2acc37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260926 14:45:34.139302 10653 leader_election.cc:302] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1733e356025d45e6a53b9234e2acc37b; no voters:
I20260926 14:45:34.139580 10653 leader_election.cc:288] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:45:34.139681 10658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:34.139879 10658 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [term 1 LEADER]: Becoming Leader. State: Replica: 1733e356025d45e6a53b9234e2acc37b, State: Running, Role: LEADER
I20260926 14:45:34.140228 10658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1733e356025d45e6a53b9234e2acc37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } }
I20260926 14:45:34.140724 10653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:45:34.141824 10660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1733e356025d45e6a53b9234e2acc37b. Latest consensus state: current_term: 1 leader_uuid: "1733e356025d45e6a53b9234e2acc37b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1733e356025d45e6a53b9234e2acc37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } } }
I20260926 14:45:34.142002 10660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:34.141829 10659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1733e356025d45e6a53b9234e2acc37b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1733e356025d45e6a53b9234e2acc37b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39383 } } }
I20260926 14:45:34.142086 10659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b [sys.catalog]: This master's current role is: LEADER
I20260926 14:45:34.142421 10668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:45:34.145319 10668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:45:34.150964 10668 catalog_manager.cc:1347] Generated new cluster ID: 1b491a838d984218aaadca0cbb46d7d2
I20260926 14:45:34.151063 10668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:45:34.166195 10668 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:45:34.167285 10668 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:45:34.173096 10668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b: Generated new TSK 0
I20260926 14:45:34.173959 10668 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:45:34.205943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:34.319960 10679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:34.320233 10679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:34.320292 10679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:34.324278 10679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:34.324338 10679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:34.324419 10679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:45:34.329593 10679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:34.330611 10679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:34.331703 10679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:34.341085 10685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:34.341087 10684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:34.341087 10687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:34.341295 10679 server_base.cc:1034] running on GCE node
I20260926 14:45:34.341688 10679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:34.342316 10679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:34.343497 10679 hybrid_clock.cc:648] HybridClock initialized: now 1790433934343470 us; error 38 us; skew 500 ppm
Sep 26 14:45:34 dist-test-slave-s7k6 krb5kdc[10572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:34.346280 10679 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:45:34.347175 10679 webserver.cc:460] Webserver started at http://127.0.98.1:36559/ using document root <none> and password file <none>
I20260926 14:45:34.347692 10679 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:34.347748 10679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:34.347927 10679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:34.349874 10679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "40ea70e9966b419ebbecccd1ae5451f7"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "cbcf31cf8f8a37f2589c07eb25bafea3"
server_key_iv: "f7384695b03cbe2e7c025996bf18a460"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.350380 10679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "40ea70e9966b419ebbecccd1ae5451f7"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "cbcf31cf8f8a37f2589c07eb25bafea3"
server_key_iv: "f7384695b03cbe2e7c025996bf18a460"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.353885 10679 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:45:34.356276 10694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:34.357584 10679 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:45:34.357683 10679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "40ea70e9966b419ebbecccd1ae5451f7"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "cbcf31cf8f8a37f2589c07eb25bafea3"
server_key_iv: "f7384695b03cbe2e7c025996bf18a460"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.357786 10679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:34.377916 10679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:34.381140 10679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:34.381311 10679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:34.381927 10679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:34.382728 10679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:34.382815 10679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:34.382874 10679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:34.382902 10679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:34.391228 10679 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46099
I20260926 14:45:34.391261 10807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46099 every 8 connection(s)
I20260926 14:45:34.392114 10679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:45:34 dist-test-slave-s7k6 krb5kdc[10572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:45:34.402295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 10679
I20260926 14:45:34.402406 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:45:34.402746 392 external_mini_cluster.cc:1428] Setting key e1e51be5a5a01dd872b62dc10f90d489
I20260926 14:45:34.405692 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.393801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46185 (local address 127.0.98.62:39383)
0926 14:45:34.393938 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:34.393943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:34.394969 (+ 1026us) server_negotiation.cc:407] Connection header received
0926 14:45:34.395809 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:34.395812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:34.395854 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:34.395913 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:34.396970 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.397516 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.398652 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.398919 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.399001 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.401593 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:34.401596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:34.401598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:34.401624 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:34.403269 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:34.403913 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:34.403916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:34.403917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:34.404055 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:34.404449 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:34.404453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:34.404454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:34.404679 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:34.404787 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:34.405404 (+ 617us) server_negotiation.cc:299] Negotiation successful
0926 14:45:34.405534 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:45:34.406535 10810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.394196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.1:46185)
0926 14:45:34.394833 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:34.394875 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:34.395631 (+ 756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:34.396065 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:34.396071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:34.396391 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:34.396842 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.396856 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.397624 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.397627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.398539 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.398548 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.399120 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.399124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.399276 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.399866 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:34.399883 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:34.401387 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:34.403408 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:34.403415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:34.403433 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:34.403795 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:34.404158 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:34.404161 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:34.404163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:34.404350 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:34.404818 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:34.404829 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:34.405154 (+ 325us) client_negotiation.cc:769] Sending connection context
0926 14:45:34.405327 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 14:45:34.405610 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":103,"threads_started":1}
I20260926 14:45:34.407919 10808 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39383
I20260926 14:45:34.408165 10808 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:34.408659 10808 heartbeater.cc:511] Master 127.0.98.62:39383 requested a full tablet report, sending...
I20260926 14:45:34.410214 10617 ts_manager.cc:194] Registered new tserver with Master: 40ea70e9966b419ebbecccd1ae5451f7 (127.0.98.1:46099)
I20260926 14:45:34.411414 10617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46185
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:45:34.429749 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:34.540345 10813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:34.540663 10813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:34.540728 10813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:34.544986 10813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:34.545053 10813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:34.545136 10813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:45:34.550527 10813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:34.551672 10813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:34.552744 10813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:34.562649 10821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:34.562593 10818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:34.562675 10819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:34.562803 10813 server_base.cc:1034] running on GCE node
I20260926 14:45:34.563406 10813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:34.564069 10813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:34.565237 10813 hybrid_clock.cc:648] HybridClock initialized: now 1790433934565203 us; error 47 us; skew 500 ppm
Sep 26 14:45:34 dist-test-slave-s7k6 krb5kdc[10572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:34.567971 10813 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:45:34.568852 10813 webserver.cc:460] Webserver started at http://127.0.98.2:36553/ using document root <none> and password file <none>
I20260926 14:45:34.569373 10813 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:34.569429 10813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:34.569589 10813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:34.571772 10813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "bbc544b66e9c4e54ad29ae11779e0053"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "fdeb8ec6c69cb168a81ebefd8c3bfd23"
server_key_iv: "fe14888f610a4117107623dcfdae1de7"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.572335 10813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bbc544b66e9c4e54ad29ae11779e0053"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "fdeb8ec6c69cb168a81ebefd8c3bfd23"
server_key_iv: "fe14888f610a4117107623dcfdae1de7"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.576237 10813 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:45:34.578817 10828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:34.580163 10813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:34.580271 10813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "bbc544b66e9c4e54ad29ae11779e0053"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "fdeb8ec6c69cb168a81ebefd8c3bfd23"
server_key_iv: "fe14888f610a4117107623dcfdae1de7"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.580365 10813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:34.611243 10813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:34.614317 10813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:34.614485 10813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:34.615092 10813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:34.615947 10813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:34.616008 10813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:34.616050 10813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:34.616071 10813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:34.624442 10813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38227
I20260926 14:45:34.624521 10941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38227 every 8 connection(s)
I20260926 14:45:34.625397 10813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:45:34.626169 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 10813
I20260926 14:45:34.626279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:45:34.626596 392 external_mini_cluster.cc:1428] Setting key d7c1a4ececb69b42823494d7a611d709
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 26 14:45:34 dist-test-slave-s7k6 krb5kdc[10572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:45:34.638942 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.627408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45173 (local address 127.0.98.62:39383)
0926 14:45:34.627528 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:34.627532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:34.628783 (+ 1251us) server_negotiation.cc:407] Connection header received
0926 14:45:34.629625 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:34.629629 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:34.629680 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:34.629757 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:34.630990 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.631565 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.632709 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.632920 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.633005 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.635539 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:34.635543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:34.635545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:34.635569 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:34.636933 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:34.637494 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:34.637498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:34.637499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:34.637576 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:34.637884 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:34.637887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:34.637889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:34.638063 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:34.638155 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:34.638641 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 14:45:34.638778 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:45:34.639731 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.627861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.2:45173)
0926 14:45:34.628617 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:34.628671 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:34.629426 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:34.629954 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:34.629961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:34.630319 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:34.630855 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.630871 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.631686 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.631689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.632588 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.632598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.633057 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.633061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.633242 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.633836 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:34.633852 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:34.635376 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:34.637082 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:34.637089 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:34.637104 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:34.637390 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:34.637659 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:34.637662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:34.637664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:34.637797 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:34.638170 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:34.638176 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:34.638418 (+ 242us) client_negotiation.cc:769] Sending connection context
0926 14:45:34.638589 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 14:45:34.638934 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":139,"threads_started":1}
I20260926 14:45:34.640812 10942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39383
I20260926 14:45:34.641072 10942 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:34.641671 10942 heartbeater.cc:511] Master 127.0.98.62:39383 requested a full tablet report, sending...
I20260926 14:45:34.642968 10617 ts_manager.cc:194] Registered new tserver with Master: bbc544b66e9c4e54ad29ae11779e0053 (127.0.98.2:38227)
I20260926 14:45:34.643736 10617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:45:34.653942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:45:34.766281 10947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:45:34.766566 10947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:45:34.766609 10947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:45:34.770732 10947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:45:34.770834 10947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:45:34.770898 10947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:45:34.776451 10947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:45:34.777536 10947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:45:34.778558 10947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:45:34.788733 10952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:34.788720 10953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:45:34.788928 10955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:45:34.788980 10947 server_base.cc:1034] running on GCE node
I20260926 14:45:34.789603 10947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:45:34.790409 10947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:45:34.791621 10947 hybrid_clock.cc:648] HybridClock initialized: now 1790433934791574 us; error 57 us; skew 500 ppm
Sep 26 14:45:34 dist-test-slave-s7k6 krb5kdc[10572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:45:34.795420 10947 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:45:34.796520 10947 webserver.cc:460] Webserver started at http://127.0.98.3:38199/ using document root <none> and password file <none>
I20260926 14:45:34.797104 10947 fs_manager.cc:359] Metadata directory not provided
I20260926 14:45:34.797173 10947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:45:34.797374 10947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:45:34.799517 10947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "49ba2199f4a5d13abf768fbf0019def1"
server_key_iv: "01155fae63ba19169eeec371afbd2b81"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.800212 10947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "49ba2199f4a5d13abf768fbf0019def1"
server_key_iv: "01155fae63ba19169eeec371afbd2b81"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.804188 10947 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:45:34.806838 10962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:34.808192 10947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:45:34.808318 10947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0"
format_stamp: "Formatted at 2026-09-26 14:45:34 on dist-test-slave-s7k6"
server_key: "49ba2199f4a5d13abf768fbf0019def1"
server_key_iv: "01155fae63ba19169eeec371afbd2b81"
server_key_version: "encryptionkey@0"
I20260926 14:45:34.808436 10947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:45:34.834740 10947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:45:34.838197 10947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:45:34.838399 10947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:45:34.839057 10947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:45:34.840008 10947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:45:34.840057 10947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:34.840109 10947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:45:34.840129 10947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:45:34.848877 10947 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36643
I20260926 14:45:34.848901 11075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36643 every 8 connection(s)
I20260926 14:45:34.849844 10947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:45:34.850477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 10947
I20260926 14:45:34.850605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:45:34.850979 392 external_mini_cluster.cc:1428] Setting key 63900bb3de8ffb10955ca5952a33f4db
Sep 26 14:45:34 dist-test-slave-s7k6 krb5kdc[10572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:45:34.863857 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.854176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51045 (local address 127.0.98.62:39383)
0926 14:45:34.854306 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:34.854309 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:34.854321 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:45:34.855023 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:34.855026 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:34.855069 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:34.855141 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:34.856292 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.856825 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.858005 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.858215 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.858289 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.860749 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:34.860752 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:34.860754 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:34.860779 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:34.862026 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:34.862536 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:34.862539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:34.862540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:34.862634 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:34.862947 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:34.862949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:34.862950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:34.863125 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:34.863199 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:34.863639 (+ 440us) server_negotiation.cc:299] Negotiation successful
0926 14:45:34.863733 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 14:45:34.864840 11078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.853067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39383 (local address 127.0.98.3:51045)
0926 14:45:34.853910 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:34.853958 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:34.854843 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:34.855285 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:34.855292 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:34.855659 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:45:34.856163 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.856177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.856959 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.856962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.857904 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.857914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.858366 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.858370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.858526 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.859156 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:45:34.859173 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:45:34.860574 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:45:34.862129 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:34.862136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:34.862150 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:34.862453 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:34.862707 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:45:34.862709 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:45:34.862711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:45:34.862877 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:45:34.863200 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:45:34.863205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:45:34.863436 (+ 231us) client_negotiation.cc:769] Sending connection context
0926 14:45:34.863579 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 14:45:34.863840 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":151,"threads_started":1}
I20260926 14:45:34.865935 11076 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39383
I20260926 14:45:34.866166 11076 heartbeater.cc:464] Registering TS with master...
I20260926 14:45:34.866658 11076 heartbeater.cc:511] Master 127.0.98.62:39383 requested a full tablet report, sending...
I20260926 14:45:34.867630 10616 ts_manager.cc:194] Registered new tserver with Master: 71965b5ec77b41f6b72e7dba3a9d4aa0 (127.0.98.3:36643)
I20260926 14:45:34.868211 10616 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51045
I20260926 14:45:34.877393 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:45:34.886346 10656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.879236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.0.98.62:39383)
0926 14:45:34.879362 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:34.879366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:34.879509 (+ 143us) server_negotiation.cc:407] Connection header received
0926 14:45:34.879615 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:34.879618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:34.879656 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:34.879718 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:45:34.880345 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.880845 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.881676 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.881878 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.881943 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.883295 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:45:34.883299 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:45:34.883302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:45:34.883328 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:45:34.884624 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:34.885170 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:34.885173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:34.885175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:34.885255 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:45:34.885522 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:45:34.885525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:45:34.885526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:45:34.885754 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:45:34.885874 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:34.886100 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 14:45:34.886196 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:45:34.889353 10616 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:45:34.892485 10616 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:45:34.906850 11088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.901448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46099 (local address 127.0.0.1:51836)
0926 14:45:34.902441 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:34.902454 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:34.902546 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:34.903055 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:34.903058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:34.903075 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:34.903170 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.903176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.904194 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.904197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.905396 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.905406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.906253 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.906256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.906393 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.906435 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 14:45:34.906543 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 14:45:34.906665 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":941,"mutex_wait_us":53,"thread_start_us":84,"threads_started":1}
I20260926 14:45:34.907193 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.901909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.0.98.2:38227)
0926 14:45:34.902336 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:34.902347 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:34.902369 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:45:34.902430 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:34.902435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:34.902632 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:34.902863 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:34.903173 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.904336 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.905695 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.906504 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.906593 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.906737 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:34.906866 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 14:45:34.907058 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":98,"threads_started":1}
I20260926 14:45:34.907235 11087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.901183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38227 (local address 127.0.0.1:55420)
0926 14:45:34.902020 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:34.902034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:34.902112 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:34.902855 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:34.902862 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:34.902886 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:34.903029 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.903035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.904478 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.904481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.905569 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.905578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.906605 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.906608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.906698 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.906719 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:45:34.906833 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 14:45:34.906906 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":1,"thread_start_us":83,"threads_started":1}
I20260926 14:45:34.907296 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.901908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51836 (local address 127.0.98.1:46099)
0926 14:45:34.902310 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:34.902326 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:34.902637 (+ 311us) server_negotiation.cc:407] Connection header received
0926 14:45:34.902715 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:34.902719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:34.902945 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:34.903074 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:34.903278 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.904067 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.905502 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.906131 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.906231 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.906389 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:34.907047 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 14:45:34.907168 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":83,"threads_started":1}
I20260926 14:45:34.909241 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.901094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36643 (local address 127.0.0.1:59070)
0926 14:45:34.901780 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:34.901835 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:34.901962 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:34.903295 (+ 1333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:34.903298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:34.903316 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:34.903391 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.903397 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.904711 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.904715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.906375 (+ 1660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.906386 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.908857 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.908860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.908947 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.908968 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:45:34.909046 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 14:45:34.909117 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":37,"thread_start_us":74,"threads_started":1}
I20260926 14:45:34.909428 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.901232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.0.98.3:36643)
0926 14:45:34.902800 (+ 1568us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:34.902811 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:34.902832 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:45:34.902899 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:34.902903 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:34.903088 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:34.903226 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:34.903508 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.904571 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.906483 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.908540 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.908658 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.908793 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:34.909099 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 14:45:34.909206 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1447,"thread_start_us":148,"threads_started":1}
I20260926 14:45:34.910322 10876 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf5c9ae5b484c12a10a246c5640d1c8 (DEFAULT_TABLE table=test-table [id=57b3c0d9f08b4f3c8ed84d13a0e032d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:34.910287 10742 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf5c9ae5b484c12a10a246c5640d1c8 (DEFAULT_TABLE table=test-table [id=57b3c0d9f08b4f3c8ed84d13a0e032d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:34.911324 10876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf5c9ae5b484c12a10a246c5640d1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:34.911342 10742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf5c9ae5b484c12a10a246c5640d1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:34.912377 11010 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf5c9ae5b484c12a10a246c5640d1c8 (DEFAULT_TABLE table=test-table [id=57b3c0d9f08b4f3c8ed84d13a0e032d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:45:34.913394 11010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf5c9ae5b484c12a10a246c5640d1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:45:34.917724 11093 tablet_bootstrap.cc:492] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053: Bootstrap starting.
I20260926 14:45:34.918864 11094 tablet_bootstrap.cc:492] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7: Bootstrap starting.
I20260926 14:45:34.920074 11095 tablet_bootstrap.cc:492] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0: Bootstrap starting.
I20260926 14:45:34.920882 11093 tablet_bootstrap.cc:654] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:34.921012 11094 tablet_bootstrap.cc:654] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:34.921783 11095 tablet_bootstrap.cc:654] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0: Neither blocks nor log segments found. Creating new log.
I20260926 14:45:34.921770 11093 log.cc:824] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:34.921770 11094 log.cc:824] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:34.922950 11095 log.cc:824] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:45:34.923866 11094 tablet_bootstrap.cc:492] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7: No bootstrap required, opened a new log
I20260926 14:45:34.923882 11093 tablet_bootstrap.cc:492] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053: No bootstrap required, opened a new log
I20260926 14:45:34.924062 11094 ts_tablet_manager.cc:1397] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:45:34.924067 11093 ts_tablet_manager.cc:1397] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260926 14:45:34.924863 11095 tablet_bootstrap.cc:492] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0: No bootstrap required, opened a new log
I20260926 14:45:34.925060 11095 ts_tablet_manager.cc:1397] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:45:34.927474 11094 raft_consensus.cc:348] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36643 } } peers { permanent_uuid: "bbc544b66e9c4e54ad29ae11779e0053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38227 } } peers { permanent_uuid: "40ea70e9966b419ebbecccd1ae5451f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46099 } }
I20260926 14:45:34.927701 11094 raft_consensus.cc:374] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:34.927753 11094 raft_consensus.cc:729] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ea70e9966b419ebbecccd1ae5451f7, State: Initialized, Role: FOLLOWER
I20260926 14:45:34.927814 11093 raft_consensus.cc:348] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36643 } } peers { permanent_uuid: "bbc544b66e9c4e54ad29ae11779e0053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38227 } } peers { permanent_uuid: "40ea70e9966b419ebbecccd1ae5451f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46099 } }
I20260926 14:45:34.928093 11093 raft_consensus.cc:374] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:34.928155 11093 raft_consensus.cc:729] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc544b66e9c4e54ad29ae11779e0053, State: Initialized, Role: FOLLOWER
I20260926 14:45:34.928205 11094 consensus_queue.cc:260] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36643 } } peers { permanent_uuid: "bbc544b66e9c4e54ad29ae11779e0053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38227 } } peers { permanent_uuid: "40ea70e9966b419ebbecccd1ae5451f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46099 } }
I20260926 14:45:34.928627 11093 consensus_queue.cc:260] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36643 } } peers { permanent_uuid: "bbc544b66e9c4e54ad29ae11779e0053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38227 } } peers { permanent_uuid: "40ea70e9966b419ebbecccd1ae5451f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46099 } }
I20260926 14:45:34.928855 10808 heartbeater.cc:503] Master 127.0.98.62:39383 was elected leader, sending a full tablet report...
I20260926 14:45:34.928777 11095 raft_consensus.cc:348] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36643 } } peers { permanent_uuid: "bbc544b66e9c4e54ad29ae11779e0053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38227 } } peers { permanent_uuid: "40ea70e9966b419ebbecccd1ae5451f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46099 } }
I20260926 14:45:34.929021 11095 raft_consensus.cc:374] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:45:34.929067 11095 raft_consensus.cc:729] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71965b5ec77b41f6b72e7dba3a9d4aa0, State: Initialized, Role: FOLLOWER
I20260926 14:45:34.929219 11094 ts_tablet_manager.cc:1428] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:45:34.929566 11093 ts_tablet_manager.cc:1428] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:45:34.929497 11095 consensus_queue.cc:260] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36643 } } peers { permanent_uuid: "bbc544b66e9c4e54ad29ae11779e0053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38227 } } peers { permanent_uuid: "40ea70e9966b419ebbecccd1ae5451f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46099 } }
I20260926 14:45:34.929890 10942 heartbeater.cc:503] Master 127.0.98.62:39383 was elected leader, sending a full tablet report...
I20260926 14:45:34.930375 11095 ts_tablet_manager.cc:1428] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:45:34.930681 11076 heartbeater.cc:503] Master 127.0.98.62:39383 was elected leader, sending a full tablet report...
I20260926 14:45:34.950860 11100 raft_consensus.cc:482] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:45:34.951049 11100 raft_consensus.cc:504] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36643 } } peers { permanent_uuid: "bbc544b66e9c4e54ad29ae11779e0053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38227 } } peers { permanent_uuid: "40ea70e9966b419ebbecccd1ae5451f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46099 } }
I20260926 14:45:34.952157 11100 leader_election.cc:288] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71965b5ec77b41f6b72e7dba3a9d4aa0 (127.0.98.3:36643), 40ea70e9966b419ebbecccd1ae5451f7 (127.0.98.1:46099)
I20260926 14:45:34.957046 11091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.952550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40345 (local address 127.0.98.1:46099)
0926 14:45:34.952666 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:34.952671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:34.952849 (+ 178us) server_negotiation.cc:407] Connection header received
0926 14:45:34.952920 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:34.952922 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:34.952971 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:34.953043 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:34.953307 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.954026 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.955779 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.956417 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.956525 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.956589 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:34.956851 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 14:45:34.956917 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:45:34.957046 11089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.952466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49247 (local address 127.0.98.3:36643)
0926 14:45:34.952571 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:34.952576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:34.952663 (+ 87us) server_negotiation.cc:407] Connection header received
0926 14:45:34.952795 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:34.952798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:34.952849 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:34.952929 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:45:34.953200 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.953977 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.955779 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.956410 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.956525 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.956589 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:34.956844 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 14:45:34.956917 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:45:34.957087 10944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.952379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36643 (local address 127.0.98.2:49247)
0926 14:45:34.952566 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:34.952580 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:34.952684 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:34.952957 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:34.952959 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:34.952972 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:34.953083 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.953090 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.954098 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.954102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.955631 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.955643 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.956537 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.956541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.956736 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.956753 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:45:34.956813 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 14:45:34.956938 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":216}
I20260926 14:45:34.957094 11102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.952393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46099 (local address 127.0.98.2:40345)
0926 14:45:34.952764 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:34.952776 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:45:34.952851 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:45:34.953054 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:45:34.953057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:45:34.953069 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:45:34.953210 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.953217 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.954130 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.954133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.955632 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:45:34.955643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.956537 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.956541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:45:34.956697 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.956737 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 14:45:34.956813 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 14:45:34.956931 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":148,"thread_start_us":115,"threads_started":1}
I20260926 14:45:34.957893 10762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf5c9ae5b484c12a10a246c5640d1c8" candidate_uuid: "bbc544b66e9c4e54ad29ae11779e0053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40ea70e9966b419ebbecccd1ae5451f7" is_pre_election: true
I20260926 14:45:34.957937 11030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf5c9ae5b484c12a10a246c5640d1c8" candidate_uuid: "bbc544b66e9c4e54ad29ae11779e0053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" is_pre_election: true
I20260926 14:45:34.958236 10762 raft_consensus.cc:2454] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbc544b66e9c4e54ad29ae11779e0053 in term 0.
I20260926 14:45:34.958235 11030 raft_consensus.cc:2454] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbc544b66e9c4e54ad29ae11779e0053 in term 0.
I20260926 14:45:34.958737 10832 leader_election.cc:302] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40ea70e9966b419ebbecccd1ae5451f7, bbc544b66e9c4e54ad29ae11779e0053; no voters:
I20260926 14:45:34.959028 11100 raft_consensus.cc:2781] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:45:34.959139 11100 raft_consensus.cc:482] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:45:34.959192 11100 raft_consensus.cc:3037] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:34.960194 11100 raft_consensus.cc:504] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36643 } } peers { permanent_uuid: "bbc544b66e9c4e54ad29ae11779e0053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38227 } } peers { permanent_uuid: "40ea70e9966b419ebbecccd1ae5451f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46099 } }
I20260926 14:45:34.960529 11100 leader_election.cc:288] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [CANDIDATE]: Term 1 election: Requested vote from peers 71965b5ec77b41f6b72e7dba3a9d4aa0 (127.0.98.3:36643), 40ea70e9966b419ebbecccd1ae5451f7 (127.0.98.1:46099)
I20260926 14:45:34.960897 11030 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf5c9ae5b484c12a10a246c5640d1c8" candidate_uuid: "bbc544b66e9c4e54ad29ae11779e0053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0"
I20260926 14:45:34.960897 10762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf5c9ae5b484c12a10a246c5640d1c8" candidate_uuid: "bbc544b66e9c4e54ad29ae11779e0053" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40ea70e9966b419ebbecccd1ae5451f7"
I20260926 14:45:34.961028 10762 raft_consensus.cc:3037] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:34.961028 11030 raft_consensus.cc:3037] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:45:34.962320 10762 raft_consensus.cc:2454] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbc544b66e9c4e54ad29ae11779e0053 in term 1.
I20260926 14:45:34.962320 11030 raft_consensus.cc:2454] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbc544b66e9c4e54ad29ae11779e0053 in term 1.
I20260926 14:45:34.962734 10830 leader_election.cc:302] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71965b5ec77b41f6b72e7dba3a9d4aa0, bbc544b66e9c4e54ad29ae11779e0053; no voters:
I20260926 14:45:34.962929 11100 raft_consensus.cc:2781] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:45:34.963155 11100 raft_consensus.cc:686] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 1 LEADER]: Becoming Leader. State: Replica: bbc544b66e9c4e54ad29ae11779e0053, State: Running, Role: LEADER
I20260926 14:45:34.963510 11100 consensus_queue.cc:237] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36643 } } peers { permanent_uuid: "bbc544b66e9c4e54ad29ae11779e0053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38227 } } peers { permanent_uuid: "40ea70e9966b419ebbecccd1ae5451f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46099 } }
I20260926 14:45:34.966529 10616 catalog_manager.cc:5495] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 reported cstate change: term changed from 0 to 1, leader changed from <none> to bbc544b66e9c4e54ad29ae11779e0053 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bbc544b66e9c4e54ad29ae11779e0053" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbc544b66e9c4e54ad29ae11779e0053" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "40ea70e9966b419ebbecccd1ae5451f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46099 } health_report { overall_health: UNKNOWN } } }
I20260926 14:45:34.987272 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:34.983490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.0.98.2:38227)
0926 14:45:34.983652 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 14:45:34.983656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:45:34.983671 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:45:34.983789 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:45:34.983791 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:45:34.983847 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:45:34.983938 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:45:34.984186 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.984798 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.985819 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:45:34.986030 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:45:34.986142 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:45:34.986391 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:45:34.986852 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:45:34.986945 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:45:34.987040 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 14:45:34.987091 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:45:34.993239 10762 raft_consensus.cc:1261] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 40ea70e9966b419ebbecccd1ae5451f7 [term 1 FOLLOWER]: Refusing update from remote peer bbc544b66e9c4e54ad29ae11779e0053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:34.993239 11030 raft_consensus.cc:1261] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0 [term 1 FOLLOWER]: Refusing update from remote peer bbc544b66e9c4e54ad29ae11779e0053: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:45:34.993994 11100 consensus_queue.cc:1035] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71965b5ec77b41f6b72e7dba3a9d4aa0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:34.994202 11103 consensus_queue.cc:1035] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40ea70e9966b419ebbecccd1ae5451f7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:45:35.002655 11107 mvcc.cc:205] Tried to move back new op lower bound from 7333617397724565504 to 7333617397612965888. Current Snapshot: MvccSnapshot[applied={T|T < 7333617397724565504}]
I20260926 14:45:35.009981 10616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:35.010205 10616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:45:35.012563 10616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1733e356025d45e6a53b9234e2acc37b: Sending DeleteTablet for 3 replicas of tablet 6cf5c9ae5b484c12a10a246c5640d1c8
I20260926 14:45:35.013341 10876 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf5c9ae5b484c12a10a246c5640d1c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55420
I20260926 14:45:35.013576 10742 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf5c9ae5b484c12a10a246c5640d1c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51836
I20260926 14:45:35.013834 11115 tablet_replica.cc:330] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053: stopping tablet replica
I20260926 14:45:35.013825 11010 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf5c9ae5b484c12a10a246c5640d1c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:45:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59070
I20260926 14:45:35.014166 11115 raft_consensus.cc:2229] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:45:35.014283 11117 tablet_replica.cc:330] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0: stopping tablet replica
I20260926 14:45:35.014532 11115 raft_consensus.cc:2258] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:35.014585 11117 raft_consensus.cc:2229] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:45:35.014928 11117 raft_consensus.cc:2258] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:45:35.015311 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 10679
I20260926 14:45:35.016706 11115 ts_tablet_manager.cc:1905] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:35.017344 11117 ts_tablet_manager.cc:1905] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:45:35.019513 11115 ts_tablet_manager.cc:1918] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:35.019614 11115 log.cc:1196] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/6cf5c9ae5b484c12a10a246c5640d1c8
I20260926 14:45:35.019927 11115 ts_tablet_manager.cc:1939] T 6cf5c9ae5b484c12a10a246c5640d1c8 P bbc544b66e9c4e54ad29ae11779e0053: Deleting consensus metadata
I20260926 14:45:35.020067 11117 ts_tablet_manager.cc:1918] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:45:35.020179 11117 log.cc:1196] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/6cf5c9ae5b484c12a10a246c5640d1c8
I20260926 14:45:35.020514 11117 ts_tablet_manager.cc:1939] T 6cf5c9ae5b484c12a10a246c5640d1c8 P 71965b5ec77b41f6b72e7dba3a9d4aa0: Deleting consensus metadata
I20260926 14:45:35.021152 10601 catalog_manager.cc:4841] TS bbc544b66e9c4e54ad29ae11779e0053 (127.0.98.2:38227): tablet 6cf5c9ae5b484c12a10a246c5640d1c8 (table test-table [id=57b3c0d9f08b4f3c8ed84d13a0e032d7]) successfully deleted
I20260926 14:45:35.021685 10602 catalog_manager.cc:4841] TS 71965b5ec77b41f6b72e7dba3a9d4aa0 (127.0.98.3:36643): tablet 6cf5c9ae5b484c12a10a246c5640d1c8 (table test-table [id=57b3c0d9f08b4f3c8ed84d13a0e032d7]) successfully deleted
W20260926 14:45:35.023461 10604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46099 (error 108)
I20260926 14:45:35.023644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 10813
I20260926 14:45:35.024801 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:45:35.024517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46099 (local address 127.0.0.1:51842)
0926 14:45:35.024660 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 14:45:35.024728 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46099: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 14:45:35.024966 10604 catalog_manager.cc:4584] TS 40ea70e9966b419ebbecccd1ae5451f7 (127.0.98.1:46099): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cf5c9ae5b484c12a10a246c5640d1c8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46099: connect: Connection refused (error 111)
I20260926 14:45:35.028851 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 10947
I20260926 14:45:35.033543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 10585
2026-09-26T14:45:35Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2165 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:45:35 dist-test-slave-s7k6 krb5kdc[11121](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:45:35 dist-test-slave-s7k6 krb5kdc[11121](info): set up 2 sockets
Sep 26 14:45:35 dist-test-slave-s7k6 krb5kdc[11121](info): commencing operation
krb5kdc: starting...
W20260926 14:45:35.983407 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:45:36 dist-test-slave-s7k6 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:45:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 14:45:36.145576 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskR2lxu1/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/postgres -l logfile start
2026-09-26 14:45:38.571 UTC [11151] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 14:45:38.571 UTC [11151] LOG: listening on IPv4 address "127.0.98.20", port 52203
2026-09-26 14:45:38.575 UTC [11151] LOG: listening on Unix socket "/tmp/.s.PGSQL.52203"
2026-09-26 14:45:38.602 UTC [11154] LOG: database system was shut down at 2026-09-26 14:45:37 UTC
2026-09-26 14:45:38.606 UTC [11151] LOG: database system is ready to accept connections
I20260926 14:45:39.475320 392 mini_postgres.cc:96] Postgres bound to 52203
2026-09-26 14:45:39.481 UTC [11162] FATAL: database "slave" does not exist
127.0.98.20:52203 - accepting connections
I20260926 14:45:39.481791 392 mini_ranger.cc:161] Starting Ranger...
I20260926 14:45:39.495887 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 14:45:39.948602 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 14:45:39.948693 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-admin
2026-09-26 14:45:40,206 [I] DB FLAVOR :POSTGRES
2026-09-26 14:45:40,207 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:45:40,207 [I] Checking connection..
2026-09-26 14:45:40,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 14:45:40,455 [I] Checking connection passed.
2026-09-26 14:45:40,455 [I] --------- Verifying version history table ---------
2026-09-26 14:45:40,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:45:40,694 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:45:40,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:45:40,957 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:45:40,957 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 14:45:40,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 14:45:41.195 UTC [11226] WARNING: there is no transaction in progress
2026-09-26 14:45:41,201 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 14:45:41,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:45:41,443 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 14:45:41,443 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:45:41,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 14:45:41,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 14:45:41,947 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s7k6.c.gcp-upstream.internal','N') ;"
2026-09-26 14:45:42,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:45:42,429 [I] Table x_portal_user does not exist in database ranger
2026-09-26 14:45:42,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:45:42,677 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 14:45:42,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:45:42,918 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:45:43,152 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 14:45:43,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 14:45:44.110 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.127 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.145 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.161 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.177 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.197 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.254 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.262 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.276 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.287 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.297 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.311 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.322 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.329 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.335 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44.342 UTC [11334] WARNING: there is no transaction in progress
2026-09-26 14:45:44,981 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 14:45:44,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 14:45:45,222 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 14:45:45,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:45:45,490 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 14:45:45,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:45:45,754 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 14:45:45,754 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:45:46,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:45:46,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s7k6.c.gcp-upstream.internal';"
2026-09-26 14:45:46,513 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 14:45:46,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 14:45:46,765 [I] DB_PATCHES have already been applied
I20260926 14:45:46.770852 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskR2lxu1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 14:45:46.770929 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 14:45:46.774214 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:44587
Sep 26, 2026 2:45:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:45:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:45:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:45:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:45:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:45:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:45:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 14:45:47.773557 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.292s user 0.000s sys 0.009s
Sep 26, 2026 2:46:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:46:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 2:46:08 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:46:08 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:46:09 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 14:46:09.690407 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 14:46:09 dist-test-slave-s7k6 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433969, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 14:46:09.769693 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 14:46:09.779215 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 14:46:10.646705 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 14:46:10.646821 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-kms
I20260926 14:46:10.646914 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:52203
2026-09-26 14:46:10,820 [I] DB FLAVOR :POSTGRES
2026-09-26 14:46:10,820 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:46:10,820 [I] Checking connection
2026-09-26 14:46:10,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 14:46:11,054 [I] connection success
2026-09-26 14:46:11,054 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 14:46:11,055 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 14:46:11,055 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 14:46:11,285 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 14:46:11,285 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:46:11,285 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 14:46:11,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:52203/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 14:46:11,562 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 14:46:12.039335 392 mini_ranger_kms.cc:310] Created kms service
I20260926 14:46:12.272043 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 14:46:12.318212 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 14:46:12 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433969, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:46:12.728749 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 14:46:12.728909 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 14:46:12.728948 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 14:46:12.732273 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11514
I20260926 14:46:12.732417 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:44217
Sep 26, 2026 2:46:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:46:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:46:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:46:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:46:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:46:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 14:46:14.802569 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.033s user 0.020s sys 0.137s
I20260926 14:46:14.802696 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 14:46:14.802811 392 mini_ranger_kms.cc:401] 127.0.98.20:44217/kms/v1/keys
Sep 26 14:46:14 dist-test-slave-s7k6 krb5kdc[11121](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790433974, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 14:46:15 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433974, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 14:46:15 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433974, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 14:46:18 dist-test-slave-s7k6 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:46:18.175050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40609
--ranger_config_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44217/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:46:18.308693 11565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:46:18.308979 11565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:46:18.309017 11565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:46:18.313076 11565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:46:18.313138 11565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:46:18.313154 11565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:46:18.313169 11565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:46:18.313184 11565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:46:18.318650 11565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:44217/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40609
--ranger_config_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:46:18.319756 11565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:46:18.320917 11565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:46:18.329767 11571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:46:18.329743 11570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:46:18.330160 11565 server_base.cc:1034] running on GCE node
W20260926 14:46:18.329720 11573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:46:18.330840 11565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:46:18.331982 11565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:46:18.333166 11565 hybrid_clock.cc:648] HybridClock initialized: now 1790433978333143 us; error 38 us; skew 500 ppm
Sep 26 14:46:18 dist-test-slave-s7k6 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:46:18.336421 11565 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:46:18.337627 11565 webserver.cc:460] Webserver started at http://127.0.98.62:34809/ using document root <none> and password file <none>
I20260926 14:46:18.338318 11565 fs_manager.cc:359] Metadata directory not provided
I20260926 14:46:18.338413 11565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:46:18.338667 11565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:46:18 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:46:18.386647 11565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "fb7cabb48af24f6d96557a966d06376f"
format_stamp: "Formatted at 2026-09-26 14:46:18 on dist-test-slave-s7k6"
server_key: "1744b8ce7b2e854343eda882f561ed20"
server_key_iv: "cff59681d61fa65b8417572be520dec7"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:18.387429 11565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb7cabb48af24f6d96557a966d06376f"
format_stamp: "Formatted at 2026-09-26 14:46:18 on dist-test-slave-s7k6"
server_key: "1744b8ce7b2e854343eda882f561ed20"
server_key_iv: "cff59681d61fa65b8417572be520dec7"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:18.392220 11565 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:46:18.410334 11580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:46:18.412021 11565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:46:18.412190 11565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "fb7cabb48af24f6d96557a966d06376f"
format_stamp: "Formatted at 2026-09-26 14:46:18 on dist-test-slave-s7k6"
server_key: "1744b8ce7b2e854343eda882f561ed20"
server_key_iv: "cff59681d61fa65b8417572be520dec7"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:18.412364 11565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:46:18.439641 11565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:46:18.443378 11565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:46:18.443599 11565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:46:18.452028 11565 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40609
I20260926 14:46:18.452051 11632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40609 every 8 connection(s)
I20260926 14:46:18.453266 11565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:46:18.456228 11633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:46:18.462296 11633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f: Bootstrap starting.
I20260926 14:46:18.463191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 11565
I20260926 14:46:18.463311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:46:18.463595 392 external_mini_cluster.cc:1428] Setting key 3d6e92e45104af6969c782a8df4bc70a
I20260926 14:46:18.465281 11633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f: Neither blocks nor log segments found. Creating new log.
I20260926 14:46:18.466197 11633 log.cc:824] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:46:18.468847 11633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f: No bootstrap required, opened a new log
I20260926 14:46:18.472761 11633 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7cabb48af24f6d96557a966d06376f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260926 14:46:18.473054 11633 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:46:18.473410 11633 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb7cabb48af24f6d96557a966d06376f, State: Initialized, Role: FOLLOWER
I20260926 14:46:18.473994 11633 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7cabb48af24f6d96557a966d06376f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260926 14:46:18.474179 11633 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:46:18.474272 11633 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:46:18.474385 11633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [term 0 FOLLOWER]: Advancing to term 1
Sep 26 14:46:18 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:46:18.475539 11633 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7cabb48af24f6d96557a966d06376f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260926 14:46:18.475896 11633 leader_election.cc:302] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb7cabb48af24f6d96557a966d06376f; no voters:
I20260926 14:46:18.476228 11633 leader_election.cc:288] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:46:18.476315 11638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:46:18.477581 11633 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:46:18.476967 11638 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [term 1 LEADER]: Becoming Leader. State: Replica: fb7cabb48af24f6d96557a966d06376f, State: Running, Role: LEADER
I20260926 14:46:18.478305 11638 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7cabb48af24f6d96557a966d06376f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } }
I20260926 14:46:18.480983 11638 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb7cabb48af24f6d96557a966d06376f. Latest consensus state: current_term: 1 leader_uuid: "fb7cabb48af24f6d96557a966d06376f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7cabb48af24f6d96557a966d06376f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } } }
I20260926 14:46:18.481146 11638 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [sys.catalog]: This master's current role is: LEADER
I20260926 14:46:18.481419 11639 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb7cabb48af24f6d96557a966d06376f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb7cabb48af24f6d96557a966d06376f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40609 } } }
I20260926 14:46:18.481511 11639 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f [sys.catalog]: This master's current role is: LEADER
I20260926 14:46:18.481586 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:18.464802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.0.98.62:40609)
0926 14:46:18.465292 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0926 14:46:18.465314 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 14:46:18.465362 (+ 48us) server_negotiation.cc:407] Connection header received
0926 14:46:18.466244 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:46:18.466270 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:46:18.466709 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:46:18.467107 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:46:18.469042 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:18.470596 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:18.471864 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:18.472608 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:18.472768 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:18.475820 (+ 3052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:46:18.475827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:46:18.475846 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:46:18.475880 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:46:18.477865 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:46:18.478376 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:46:18.478383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:46:18.478390 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:46:18.478570 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:46:18.478896 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:46:18.478900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:46:18.478901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:46:18.479458 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:46:18.479665 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0926 14:46:18.480051 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 14:46:18.480376 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":149,"threads_started":1}
I20260926 14:46:18.482158 11633 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 14:46:18.483055 11652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:46:18.489490 11652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:46:18.498814 11652 catalog_manager.cc:1347] Generated new cluster ID: 49012520da004841943c8ab71bf42b12
I20260926 14:46:18.498982 11652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:46:18.538028 11652 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:46:18.539539 11652 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:46:18.553675 11652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f: Generated new TSK 0
I20260926 14:46:18.555789 11652 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:46:19.274384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:46:19.418378 11678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:46:19.418659 11678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:46:19.418720 11678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:46:19.423097 11678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:46:19.423166 11678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:46:19.423249 11678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:46:19.428930 11678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:44217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:46:19.430559 11678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:46:19.432168 11678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:46:19.442184 11685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:46:19.442184 11688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:46:19.442698 11686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:46:19.442873 11678 server_base.cc:1034] running on GCE node
I20260926 14:46:19.443382 11678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:46:19.444092 11678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:46:19.445252 11678 hybrid_clock.cc:648] HybridClock initialized: now 1790433979445228 us; error 38 us; skew 500 ppm
Sep 26 14:46:19 dist-test-slave-s7k6 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:46:19.448228 11678 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:46:19.449193 11678 webserver.cc:460] Webserver started at http://127.0.98.1:46819/ using document root <none> and password file <none>
I20260926 14:46:19.449766 11678 fs_manager.cc:359] Metadata directory not provided
I20260926 14:46:19.449836 11678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:46:19.450059 11678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:46:19 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:46:19.474800 11678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "f3c9189b2cf94f8fb8b0a9332deaa547"
format_stamp: "Formatted at 2026-09-26 14:46:19 on dist-test-slave-s7k6"
server_key: "4f26173ce44191e79dce1e7320e01c2a"
server_key_iv: "5d9052f4fc8e77cb38c6329c6103ca3b"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:19.475692 11678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f3c9189b2cf94f8fb8b0a9332deaa547"
format_stamp: "Formatted at 2026-09-26 14:46:19 on dist-test-slave-s7k6"
server_key: "4f26173ce44191e79dce1e7320e01c2a"
server_key_iv: "5d9052f4fc8e77cb38c6329c6103ca3b"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:19.483175 11678 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260926 14:46:19.504572 11695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:46:19.506366 11678 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 14:46:19.506513 11678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "f3c9189b2cf94f8fb8b0a9332deaa547"
format_stamp: "Formatted at 2026-09-26 14:46:19 on dist-test-slave-s7k6"
server_key: "4f26173ce44191e79dce1e7320e01c2a"
server_key_iv: "5d9052f4fc8e77cb38c6329c6103ca3b"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:19.506675 11678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:46:19.529999 11678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:46:19.539741 11678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:46:19.540023 11678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:46:19.540804 11678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:46:19.541929 11678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:46:19.542035 11678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:46:19.542160 11678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:46:19.542239 11678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:46:19.556828 11678 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43315
I20260926 14:46:19.558135 11678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:46:19.559934 11810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43315 every 8 connection(s)
I20260926 14:46:19.564323 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 11678
I20260926 14:46:19.564456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:46:19.564819 392 external_mini_cluster.cc:1428] Setting key 650c3d16ce6bbbcdb7e434590aca3600
Sep 26 14:46:19 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 14:46:19.582091 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:19.565114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35383 (local address 127.0.98.62:40609)
0926 14:46:19.567757 (+ 2643us) server_negotiation.cc:206] Beginning negotiation
0926 14:46:19.567762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:46:19.567777 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:46:19.567817 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:46:19.567820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:46:19.567869 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:46:19.567954 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:46:19.569644 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:19.570595 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:19.573022 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:19.573449 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:19.573563 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:19.576870 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:46:19.576874 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:46:19.576877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:46:19.576909 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:46:19.579113 (+ 2204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:46:19.579875 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:46:19.579880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:46:19.579882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:46:19.579981 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:46:19.580385 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:46:19.580388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:46:19.580390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:46:19.580629 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:46:19.580720 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:46:19.581497 (+ 777us) server_negotiation.cc:299] Negotiation successful
0926 14:46:19.581647 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2556,"thread_start_us":87,"threads_started":1}
I20260926 14:46:19.583192 11813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:19.562690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40609 (local address 127.0.98.1:35383)
0926 14:46:19.565928 (+ 3238us) negotiation.cc:107] Waiting for socket to connect
0926 14:46:19.565994 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 14:46:19.567283 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:46:19.568304 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:46:19.568316 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:46:19.568684 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:46:19.569466 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:19.569488 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:19.570780 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:19.570784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:19.572839 (+ 2055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:19.572856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:19.573775 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:19.573779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:19.574061 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:19.574867 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:46:19.574891 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:46:19.576619 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:46:19.579322 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:46:19.579330 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:46:19.579350 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:46:19.579752 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:46:19.580074 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:46:19.580077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:46:19.580079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:46:19.580282 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:46:19.580790 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:46:19.580798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:46:19.581168 (+ 370us) client_negotiation.cc:769] Sending connection context
0926 14:46:19.581383 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 14:46:19.581747 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2881,"thread_start_us":186,"threads_started":1}
I20260926 14:46:19.585054 11811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40609
I20260926 14:46:19.585342 11811 heartbeater.cc:464] Registering TS with master...
I20260926 14:46:19.585966 11811 heartbeater.cc:511] Master 127.0.98.62:40609 requested a full tablet report, sending...
I20260926 14:46:19.587965 11597 ts_manager.cc:194] Registered new tserver with Master: f3c9189b2cf94f8fb8b0a9332deaa547 (127.0.98.1:43315)
I20260926 14:46:19.589617 11597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:46:19.599690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:46:19 dist-test-slave-s7k6 krb5kdc[11121](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790433979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 14:46:19.736675 11817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:46:19.737023 11817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:46:19.737110 11817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:46:19.741361 11817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:46:19.741425 11817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:46:19.741505 11817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:46:19.746922 11817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:44217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:46:19.747954 11817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:46:19.748956 11817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:46:19.760252 11826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:46:19.760492 11823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:46:19.760236 11824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:46:19.762046 11817 server_base.cc:1034] running on GCE node
I20260926 14:46:19.762547 11817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:46:19.763309 11817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:46:19.764484 11817 hybrid_clock.cc:648] HybridClock initialized: now 1790433979764455 us; error 42 us; skew 500 ppm
Sep 26 14:46:19 dist-test-slave-s7k6 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:46:19.767357 11817 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:46:19.768352 11817 webserver.cc:460] Webserver started at http://127.0.98.2:37301/ using document root <none> and password file <none>
I20260926 14:46:19.768925 11817 fs_manager.cc:359] Metadata directory not provided
I20260926 14:46:19.769021 11817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:46:19.769284 11817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:46:19 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:46:19.788246 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "020790ea74434b04bf9cdaaf8f6f872a"
format_stamp: "Formatted at 2026-09-26 14:46:19 on dist-test-slave-s7k6"
server_key: "84e47d84f351cb50b6d862f292cfddcc"
server_key_iv: "df16d6f5ff2217fa2a88b6da5efd2432"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:19.789121 11817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "020790ea74434b04bf9cdaaf8f6f872a"
format_stamp: "Formatted at 2026-09-26 14:46:19 on dist-test-slave-s7k6"
server_key: "84e47d84f351cb50b6d862f292cfddcc"
server_key_iv: "df16d6f5ff2217fa2a88b6da5efd2432"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:19.796295 11817 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260926 14:46:19.824470 11833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:46:19.831125 11817 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.000s
I20260926 14:46:19.831331 11817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "020790ea74434b04bf9cdaaf8f6f872a"
format_stamp: "Formatted at 2026-09-26 14:46:19 on dist-test-slave-s7k6"
server_key: "84e47d84f351cb50b6d862f292cfddcc"
server_key_iv: "df16d6f5ff2217fa2a88b6da5efd2432"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:19.831501 11817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:46:19.871801 11817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:46:19.875641 11817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:46:19.875906 11817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:46:19.876633 11817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:46:19.877731 11817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:46:19.877848 11817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:46:19.877943 11817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:46:19.878003 11817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:46:19.889663 11817 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34095
I20260926 14:46:19.889716 11946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34095 every 8 connection(s)
I20260926 14:46:19.890619 11817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:46:19.893450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 11817
I20260926 14:46:19.893577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:46:19.893878 392 external_mini_cluster.cc:1428] Setting key aece57aed97be17a9cf248d8b8e5f7e6
Sep 26 14:46:19 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 14:46:19.906330 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:19.892546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50281 (local address 127.0.98.62:40609)
0926 14:46:19.892699 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 14:46:19.892704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:46:19.893661 (+ 957us) server_negotiation.cc:407] Connection header received
0926 14:46:19.894936 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:46:19.894939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:46:19.894993 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:46:19.895042 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:46:19.896813 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:19.897627 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:19.898926 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:19.899256 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:19.899355 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:19.902150 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:46:19.902154 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:46:19.902157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:46:19.902192 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:46:19.903954 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:46:19.904602 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:46:19.904606 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:46:19.904608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:46:19.904701 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:46:19.905112 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:46:19.905116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:46:19.905118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:46:19.905305 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:46:19.905392 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:46:19.906007 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 14:46:19.906156 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:46:19.907446 11949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:19.892954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40609 (local address 127.0.98.2:50281)
0926 14:46:19.893477 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 14:46:19.893539 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 14:46:19.894648 (+ 1109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:46:19.895482 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:46:19.895492 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:46:19.895858 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:46:19.896557 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:19.896595 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:19.897786 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:19.897789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:19.898782 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:19.898795 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:19.899400 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:19.899403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:19.899556 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:19.900178 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:46:19.900201 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:46:19.901924 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:46:19.904103 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:46:19.904110 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:46:19.904129 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:46:19.904473 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:46:19.904824 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:46:19.904827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:46:19.904829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:46:19.905016 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:46:19.905407 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:46:19.905414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:46:19.905767 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 14:46:19.905947 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 14:46:19.906287 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":115,"threads_started":1}
I20260926 14:46:19.908968 11947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40609
I20260926 14:46:19.909273 11947 heartbeater.cc:464] Registering TS with master...
I20260926 14:46:19.909947 11947 heartbeater.cc:511] Master 127.0.98.62:40609 requested a full tablet report, sending...
I20260926 14:46:19.911288 11597 ts_manager.cc:194] Registered new tserver with Master: 020790ea74434b04bf9cdaaf8f6f872a (127.0.98.2:34095)
I20260926 14:46:19.911862 11597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50281
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:46:19.927408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:44217/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:46:20.066798 11952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:46:20.067178 11952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:46:20.067237 11952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:46:20.073593 11952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:46:20.073762 11952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:46:20.073983 11952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:46:20.082365 11952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:44217/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:46:20.084218 11952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:46:20.085749 11952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 26 14:46:20 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 14:46:20.099553 11958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:46:20.099611 11960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:46:20.100064 11957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:46:20.101297 11952 server_base.cc:1034] running on GCE node
I20260926 14:46:20.101818 11952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:46:20.102662 11952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:46:20.105986 11952 hybrid_clock.cc:648] HybridClock initialized: now 1790433980105927 us; error 69 us; skew 500 ppm
Sep 26 14:46:20 dist-test-slave-s7k6 krb5kdc[11121](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:46:20.109967 11952 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:46:20.111205 11952 webserver.cc:460] Webserver started at http://127.0.98.3:38873/ using document root <none> and password file <none>
I20260926 14:46:20.111979 11952 fs_manager.cc:359] Metadata directory not provided
I20260926 14:46:20.112128 11952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:46:20.112406 11952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:46:20 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:46:20.150337 11952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca"
format_stamp: "Formatted at 2026-09-26 14:46:20 on dist-test-slave-s7k6"
server_key: "510eb5a6bc0b1e7d3acc6c20b4c1bbd7"
server_key_iv: "516ea16b0100247bcf2037d256c5e4a3"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:20.153853 11952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca"
format_stamp: "Formatted at 2026-09-26 14:46:20 on dist-test-slave-s7k6"
server_key: "510eb5a6bc0b1e7d3acc6c20b4c1bbd7"
server_key_iv: "516ea16b0100247bcf2037d256c5e4a3"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:20.180603 11952 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.005s sys 0.000s
I20260926 14:46:20.212776 11967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:46:20.214780 11952 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 14:46:20.215000 11952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca"
format_stamp: "Formatted at 2026-09-26 14:46:20 on dist-test-slave-s7k6"
server_key: "510eb5a6bc0b1e7d3acc6c20b4c1bbd7"
server_key_iv: "516ea16b0100247bcf2037d256c5e4a3"
server_key_version: "kuduclusterkey@0"
I20260926 14:46:20.215243 11952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:46:20.238847 11952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:46:20.242734 11952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:46:20.243016 11952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:46:20.243789 11952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:46:20.247076 11952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:46:20.247159 11952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:46:20.247216 11952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:46:20.247277 11952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:46:20.258543 11952 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41383
I20260926 14:46:20.258682 12080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41383 every 8 connection(s)
I20260926 14:46:20.259961 11952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:46:20.260651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 11952
I20260926 14:46:20.260785 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:46:20.261098 392 external_mini_cluster.cc:1428] Setting key 7b249f8c9621345710e6460a9eeb91fd
Sep 26 14:46:20 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:46:20.284510 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.265930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53293 (local address 127.0.98.62:40609)
0926 14:46:20.267013 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0926 14:46:20.267018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:46:20.270131 (+ 3113us) server_negotiation.cc:407] Connection header received
0926 14:46:20.271948 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:46:20.271952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:46:20.272010 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:46:20.272106 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:46:20.273577 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.274409 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.276111 (+ 1702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.276462 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.276567 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.279725 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:46:20.279729 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:46:20.279732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:46:20.279763 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:46:20.281659 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:46:20.282518 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:46:20.282523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:46:20.282525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:46:20.282624 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:46:20.283126 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:46:20.283129 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:46:20.283130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:46:20.283362 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:46:20.283452 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:46:20.284215 (+ 763us) server_negotiation.cc:299] Negotiation successful
0926 14:46:20.284351 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785}
I20260926 14:46:20.285894 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.263419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40609 (local address 127.0.98.3:53293)
0926 14:46:20.269677 (+ 6258us) negotiation.cc:107] Waiting for socket to connect
0926 14:46:20.269768 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0926 14:46:20.271667 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:46:20.272347 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:46:20.272358 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:46:20.272707 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:46:20.273406 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.273424 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.274553 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.274557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.275976 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.275988 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.276621 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.276624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.276827 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.277532 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:46:20.277552 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:46:20.279479 (+ 1927us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:46:20.281943 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:46:20.281951 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:46:20.281972 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:46:20.282390 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:46:20.282741 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:46:20.282744 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:46:20.282746 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:46:20.283013 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:46:20.283477 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:46:20.283490 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:46:20.283923 (+ 433us) client_negotiation.cc:769] Sending connection context
0926 14:46:20.284166 (+ 243us) client_negotiation.cc:240] Negotiation successful
0926 14:46:20.284552 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5896,"thread_start_us":149,"threads_started":1}
I20260926 14:46:20.287235 12081 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40609
I20260926 14:46:20.287503 12081 heartbeater.cc:464] Registering TS with master...
I20260926 14:46:20.288122 12081 heartbeater.cc:511] Master 127.0.98.62:40609 requested a full tablet report, sending...
I20260926 14:46:20.289320 11597 ts_manager.cc:194] Registered new tserver with Master: 4dfda22ed5374cdaa1836ad0ab6f72ca (127.0.98.3:41383)
I20260926 14:46:20.290107 11597 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53293
I20260926 14:46:20.293558 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:46:20.305420 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.296005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56788 (local address 127.0.98.62:40609)
0926 14:46:20.296289 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0926 14:46:20.296295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:46:20.296342 (+ 47us) server_negotiation.cc:407] Connection header received
0926 14:46:20.296505 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:46:20.296508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:46:20.296557 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:46:20.296640 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:46:20.297502 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.298309 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.299558 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.299905 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.300010 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.301706 (+ 1696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:46:20.301710 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:46:20.301713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:46:20.301743 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:46:20.303482 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:46:20.303993 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:46:20.303996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:46:20.303998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:46:20.304087 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:46:20.304380 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:46:20.304383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:46:20.304385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:46:20.304601 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:46:20.304724 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 14:46:20.304996 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 14:46:20.305110 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200}
I20260926 14:46:20.308802 11597 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:46:20.313606 11597 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:46:20.336768 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.324899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.0.1:42594)
0926 14:46:20.327045 (+ 2146us) negotiation.cc:107] Waiting for socket to connect
0926 14:46:20.327105 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 14:46:20.327247 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:46:20.328056 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:46:20.328063 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:46:20.328085 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:46:20.328218 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.328226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.330480 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.330484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.334983 (+ 4499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.334994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.336258 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.336261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.336400 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.336424 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:46:20.336517 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 14:46:20.336598 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1954,"mutex_wait_us":2070,"thread_start_us":104,"threads_started":1}
I20260926 14:46:20.337328 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.325765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48378 (local address 127.0.98.1:43315)
0926 14:46:20.327680 (+ 1915us) server_negotiation.cc:206] Beginning negotiation
0926 14:46:20.327686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:46:20.327707 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:46:20.327779 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:46:20.327783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:46:20.327980 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:46:20.328114 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:46:20.328494 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.330699 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.334265 (+ 3566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.335111 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.335220 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.335377 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 14:46:20.337071 (+ 1694us) server_negotiation.cc:299] Negotiation successful
0926 14:46:20.337194 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1813,"thread_start_us":99,"threads_started":1}
I20260926 14:46:20.337682 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.326239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42594 (local address 127.0.98.2:34095)
0926 14:46:20.327046 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0926 14:46:20.327062 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:46:20.327392 (+ 330us) server_negotiation.cc:407] Connection header received
0926 14:46:20.327477 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:46:20.327482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:46:20.327718 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:46:20.327887 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:46:20.328346 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.330338 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.335142 (+ 4804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.335964 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.336065 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.336197 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:46:20.337162 (+ 965us) server_negotiation.cc:299] Negotiation successful
0926 14:46:20.337272 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":64,"threads_started":1}
I20260926 14:46:20.336954 12096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.325112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43315 (local address 127.0.0.1:48378)
0926 14:46:20.327348 (+ 2236us) negotiation.cc:107] Waiting for socket to connect
0926 14:46:20.327361 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:46:20.327455 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:46:20.328251 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:46:20.328254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:46:20.328270 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:46:20.328363 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.328369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.332992 (+ 4623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.332995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.334111 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.334122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.335443 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.335447 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.335569 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.335612 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 14:46:20.335727 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 14:46:20.336816 (+ 1089us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2175,"thread_start_us":109,"threads_started":1}
I20260926 14:46:20.338311 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.325833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41383 (local address 127.0.0.1:53578)
0926 14:46:20.329192 (+ 3359us) negotiation.cc:107] Waiting for socket to connect
0926 14:46:20.329205 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:46:20.329289 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:46:20.330859 (+ 1570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:46:20.330862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:46:20.330884 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:46:20.330976 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.330982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.332362 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.332366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.333598 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.333609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.337927 (+ 4318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.337931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.338020 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.338043 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:46:20.338114 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 14:46:20.338186 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3298,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260926 14:46:20.338840 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.326532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53578 (local address 127.0.98.3:41383)
0926 14:46:20.327216 (+ 684us) server_negotiation.cc:206] Beginning negotiation
0926 14:46:20.327226 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:46:20.329374 (+ 2148us) server_negotiation.cc:407] Connection header received
0926 14:46:20.329452 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:46:20.329456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:46:20.329632 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:46:20.329796 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:46:20.331097 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.332229 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.333756 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.334600 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.334702 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.334910 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 14:46:20.338535 (+ 3625us) server_negotiation.cc:299] Negotiation successful
0926 14:46:20.338653 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":129,"threads_started":1}
I20260926 14:46:20.341239 11745 tablet_service.cc:1467] Processing CreateTablet for tablet 747b1af12d654817b6e45047037ab84f (DEFAULT_TABLE table=test-table [id=4c439dcea0d64686a5ddebf691c5323c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:46:20.342542 11745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747b1af12d654817b6e45047037ab84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:46:20.342980 12015 tablet_service.cc:1467] Processing CreateTablet for tablet 747b1af12d654817b6e45047037ab84f (DEFAULT_TABLE table=test-table [id=4c439dcea0d64686a5ddebf691c5323c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:46:20.344059 12015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747b1af12d654817b6e45047037ab84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:46:20.345018 11880 tablet_service.cc:1467] Processing CreateTablet for tablet 747b1af12d654817b6e45047037ab84f (DEFAULT_TABLE table=test-table [id=4c439dcea0d64686a5ddebf691c5323c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:46:20.346035 11880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747b1af12d654817b6e45047037ab84f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:46:20.351763 12099 tablet_bootstrap.cc:492] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547: Bootstrap starting.
I20260926 14:46:20.354425 12099 tablet_bootstrap.cc:654] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547: Neither blocks nor log segments found. Creating new log.
I20260926 14:46:20.355247 12100 tablet_bootstrap.cc:492] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: Bootstrap starting.
I20260926 14:46:20.355482 12099 log.cc:824] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547: Log is configured to *not* fsync() on all Append() calls
I20260926 14:46:20.357910 12099 tablet_bootstrap.cc:492] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547: No bootstrap required, opened a new log
I20260926 14:46:20.358109 12099 ts_tablet_manager.cc:1397] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 14:46:20.358150 12100 tablet_bootstrap.cc:654] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: Neither blocks nor log segments found. Creating new log.
I20260926 14:46:20.359022 12100 log.cc:824] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: Log is configured to *not* fsync() on all Append() calls
I20260926 14:46:20.360423 12101 tablet_bootstrap.cc:492] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a: Bootstrap starting.
I20260926 14:46:20.363073 12099 raft_consensus.cc:348] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020790ea74434b04bf9cdaaf8f6f872a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41383 } } peers { permanent_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43315 } }
I20260926 14:46:20.363535 12099 raft_consensus.cc:374] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:46:20.363608 12101 tablet_bootstrap.cc:654] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a: Neither blocks nor log segments found. Creating new log.
I20260926 14:46:20.363638 12099 raft_consensus.cc:729] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c9189b2cf94f8fb8b0a9332deaa547, State: Initialized, Role: FOLLOWER
I20260926 14:46:20.364238 12099 consensus_queue.cc:260] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020790ea74434b04bf9cdaaf8f6f872a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41383 } } peers { permanent_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43315 } }
I20260926 14:46:20.365069 12101 log.cc:824] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:46:20.365526 12099 ts_tablet_manager.cc:1428] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 14:46:20.365988 11811 heartbeater.cc:503] Master 127.0.98.62:40609 was elected leader, sending a full tablet report...
I20260926 14:46:20.368369 12100 tablet_bootstrap.cc:492] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: No bootstrap required, opened a new log
I20260926 14:46:20.368606 12100 ts_tablet_manager.cc:1397] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260926 14:46:20.372856 12100 raft_consensus.cc:348] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020790ea74434b04bf9cdaaf8f6f872a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41383 } } peers { permanent_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43315 } }
I20260926 14:46:20.373188 12100 raft_consensus.cc:374] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:46:20.373250 12100 raft_consensus.cc:729] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dfda22ed5374cdaa1836ad0ab6f72ca, State: Initialized, Role: FOLLOWER
I20260926 14:46:20.373771 12100 consensus_queue.cc:260] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020790ea74434b04bf9cdaaf8f6f872a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41383 } } peers { permanent_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43315 } }
I20260926 14:46:20.374537 12081 heartbeater.cc:503] Master 127.0.98.62:40609 was elected leader, sending a full tablet report...
I20260926 14:46:20.375021 12100 ts_tablet_manager.cc:1428] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260926 14:46:20.376791 12101 tablet_bootstrap.cc:492] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a: No bootstrap required, opened a new log
I20260926 14:46:20.377012 12101 ts_tablet_manager.cc:1397] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.000s
I20260926 14:46:20.381727 12101 raft_consensus.cc:348] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020790ea74434b04bf9cdaaf8f6f872a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41383 } } peers { permanent_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43315 } }
I20260926 14:46:20.382046 12101 raft_consensus.cc:374] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:46:20.382093 12101 raft_consensus.cc:729] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 020790ea74434b04bf9cdaaf8f6f872a, State: Initialized, Role: FOLLOWER
I20260926 14:46:20.382656 12101 consensus_queue.cc:260] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020790ea74434b04bf9cdaaf8f6f872a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41383 } } peers { permanent_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43315 } }
I20260926 14:46:20.383481 11947 heartbeater.cc:503] Master 127.0.98.62:40609 was elected leader, sending a full tablet report...
I20260926 14:46:20.385001 12101 ts_tablet_manager.cc:1428] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
W20260926 14:46:20.392014 11948 tablet.cc:2369] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 14:46:20 dist-test-slave-s7k6 krb5kdc[11121](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433979, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:46:20.447283 12106 raft_consensus.cc:482] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:46:20.447546 12106 raft_consensus.cc:504] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020790ea74434b04bf9cdaaf8f6f872a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41383 } } peers { permanent_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43315 } }
I20260926 14:46:20.449092 12106 leader_election.cc:288] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 020790ea74434b04bf9cdaaf8f6f872a (127.0.98.2:34095), f3c9189b2cf94f8fb8b0a9332deaa547 (127.0.98.1:43315)
I20260926 14:46:20.454607 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.449276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.98.3:60095)
0926 14:46:20.449590 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 14:46:20.449606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:46:20.449700 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:46:20.450373 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:46:20.450377 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:46:20.450396 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:46:20.450521 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.450528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.451667 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.451671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.452995 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.453007 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.454085 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.454089 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.454281 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.454306 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:46:20.454379 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 14:46:20.454462 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":18}
I20260926 14:46:20.454655 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.449877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60095 (local address 127.0.98.2:34095)
0926 14:46:20.450044 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 14:46:20.450049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:46:20.450066 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:46:20.450122 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:46:20.450125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:46:20.450175 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:46:20.450252 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:46:20.450670 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.451514 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.453199 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.453823 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.453952 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.454022 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:46:20.454432 (+ 410us) server_negotiation.cc:299] Negotiation successful
0926 14:46:20.454524 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 14:46:20.455235 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.449675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42119 (local address 127.0.98.1:43315)
0926 14:46:20.449795 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:46:20.449799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:46:20.450210 (+ 411us) server_negotiation.cc:407] Connection header received
0926 14:46:20.450403 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:46:20.450407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:46:20.450458 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:46:20.450504 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:46:20.450982 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.451758 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.453195 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.453821 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.453897 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.453967 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:46:20.455037 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0926 14:46:20.455119 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:46:20.455353 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747b1af12d654817b6e45047037ab84f" candidate_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "020790ea74434b04bf9cdaaf8f6f872a" is_pre_election: true
I20260926 14:46:20.455646 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.449284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43315 (local address 127.0.98.3:42119)
0926 14:46:20.450125 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0926 14:46:20.450138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:46:20.450282 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:46:20.450663 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:46:20.450666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:46:20.450683 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:46:20.450853 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.450860 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.451903 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.451907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.453061 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:46:20.453071 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.454067 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.454070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:46:20.454905 (+ 835us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.454931 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:46:20.455415 (+ 484us) client_negotiation.cc:240] Negotiation successful
0926 14:46:20.455493 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":87,"thread_start_us":108,"threads_started":1}
I20260926 14:46:20.455719 11901 raft_consensus.cc:2454] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dfda22ed5374cdaa1836ad0ab6f72ca in term 0.
I20260926 14:46:20.456205 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747b1af12d654817b6e45047037ab84f" candidate_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" is_pre_election: true
I20260926 14:46:20.456481 11765 raft_consensus.cc:2454] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4dfda22ed5374cdaa1836ad0ab6f72ca in term 0.
I20260926 14:46:20.456790 11969 leader_election.cc:302] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 020790ea74434b04bf9cdaaf8f6f872a, 4dfda22ed5374cdaa1836ad0ab6f72ca; no voters:
I20260926 14:46:20.457388 12106 raft_consensus.cc:2781] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:46:20.457490 12106 raft_consensus.cc:482] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:46:20.457543 12106 raft_consensus.cc:3037] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:46:20.458972 12106 raft_consensus.cc:504] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020790ea74434b04bf9cdaaf8f6f872a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41383 } } peers { permanent_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43315 } }
I20260926 14:46:20.459779 11765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747b1af12d654817b6e45047037ab84f" candidate_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547"
I20260926 14:46:20.459929 11765 raft_consensus.cc:3037] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:46:20.461333 11765 raft_consensus.cc:2454] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dfda22ed5374cdaa1836ad0ab6f72ca in term 1.
I20260926 14:46:20.461699 11970 leader_election.cc:302] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4dfda22ed5374cdaa1836ad0ab6f72ca, f3c9189b2cf94f8fb8b0a9332deaa547; no voters:
I20260926 14:46:20.461995 11901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747b1af12d654817b6e45047037ab84f" candidate_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "020790ea74434b04bf9cdaaf8f6f872a"
I20260926 14:46:20.462133 11901 raft_consensus.cc:3037] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:46:20.463428 11901 raft_consensus.cc:2454] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4dfda22ed5374cdaa1836ad0ab6f72ca in term 1.
I20260926 14:46:20.464152 12106 leader_election.cc:288] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [CANDIDATE]: Term 1 election: Requested vote from peers 020790ea74434b04bf9cdaaf8f6f872a (127.0.98.2:34095), f3c9189b2cf94f8fb8b0a9332deaa547 (127.0.98.1:43315)
I20260926 14:46:20.464330 12106 raft_consensus.cc:2781] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:46:20.464425 12106 raft_consensus.cc:686] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 1 LEADER]: Becoming Leader. State: Replica: 4dfda22ed5374cdaa1836ad0ab6f72ca, State: Running, Role: LEADER
I20260926 14:46:20.464802 12106 consensus_queue.cc:237] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020790ea74434b04bf9cdaaf8f6f872a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } } peers { permanent_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41383 } } peers { permanent_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43315 } }
I20260926 14:46:20.468461 11592 catalog_manager.cc:5495] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 4dfda22ed5374cdaa1836ad0ab6f72ca (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "020790ea74434b04bf9cdaaf8f6f872a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4dfda22ed5374cdaa1836ad0ab6f72ca" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43315 } health_report { overall_health: UNKNOWN } } }
I20260926 14:46:20.501542 12098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.497540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53582 (local address 127.0.98.3:41383)
0926 14:46:20.497690 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:46:20.497695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:46:20.497787 (+ 92us) server_negotiation.cc:407] Connection header received
0926 14:46:20.497958 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:46:20.497960 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:46:20.498016 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:46:20.498113 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:46:20.498357 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.498929 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.499800 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:46:20.500068 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:46:20.500158 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:46:20.500485 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:46:20.501103 (+ 618us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:46:20.501233 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 14:46:20.501326 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 14:46:20.501385 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:46:20.508061 11901 raft_consensus.cc:1261] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a [term 1 FOLLOWER]: Refusing update from remote peer 4dfda22ed5374cdaa1836ad0ab6f72ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:46:20.508062 11765 raft_consensus.cc:1261] T 747b1af12d654817b6e45047037ab84f P f3c9189b2cf94f8fb8b0a9332deaa547 [term 1 FOLLOWER]: Refusing update from remote peer 4dfda22ed5374cdaa1836ad0ab6f72ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:46:20.508880 12110 consensus_queue.cc:1035] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "020790ea74434b04bf9cdaaf8f6f872a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:46:20.509135 12106 consensus_queue.cc:1035] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3c9189b2cf94f8fb8b0a9332deaa547" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:46:20.512574 12082 tablet.cc:2369] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:46:20.532150 12115 mvcc.cc:205] Tried to move back new op lower bound from 7333617584152096768 to 7333617583986974720. Current Snapshot: MvccSnapshot[applied={T|T < 7333617584152096768 or (T in {7333617584152096768})}]
I20260926 14:46:20.532526 11592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:46:20.532697 11592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:46:20.533200 12117 mvcc.cc:205] Tried to move back new op lower bound from 7333617584152096768 to 7333617583986974720. Current Snapshot: MvccSnapshot[applied={T|T < 7333617584152096768}]
I20260926 14:46:20.535542 11592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb7cabb48af24f6d96557a966d06376f: Sending DeleteTablet for 3 replicas of tablet 747b1af12d654817b6e45047037ab84f
I20260926 14:46:20.536471 11745 tablet_service.cc:1514] Processing DeleteTablet for tablet 747b1af12d654817b6e45047037ab84f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:46:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48378
I20260926 14:46:20.536551 12015 tablet_service.cc:1514] Processing DeleteTablet for tablet 747b1af12d654817b6e45047037ab84f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:46:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53578
I20260926 14:46:20.537659 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 11678
I20260926 14:46:20.537402 11880 tablet_service.cc:1514] Processing DeleteTablet for tablet 747b1af12d654817b6e45047037ab84f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:46:20 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42594
I20260926 14:46:20.543150 12125 tablet_replica.cc:330] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a: stopping tablet replica
I20260926 14:46:20.543818 12125 raft_consensus.cc:2229] T 747b1af12d654817b6e45047037ab84f P 020790ea74434b04bf9cdaaf8f6f872a [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 14:46:20.549870 11583 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43315 (error 108)
I20260926 14:46:20.550110 12124 tablet_replica.cc:330] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: stopping tablet replica
I20260926 14:46:20.550475 12124 raft_consensus.cc:2229] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:46:20.550948 12124 raft_consensus.cc:2258] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:46:20.551602 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:46:20.551247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43315 (local address 127.0.0.1:48394)
0926 14:46:20.551440 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 14:46:20.551513 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260926 14:46:20.551888 11583 catalog_manager.cc:4584] TS f3c9189b2cf94f8fb8b0a9332deaa547 (127.0.98.1:43315): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 747b1af12d654817b6e45047037ab84f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43315: connect: Connection refused (error 111)
I20260926 14:46:20.552255 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 11817
I20260926 14:46:20.552480 12124 ts_tablet_manager.cc:1905] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:46:20.555460 12124 ts_tablet_manager.cc:1918] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:46:20.555567 12124 log.cc:1196] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/747b1af12d654817b6e45047037ab84f
I20260926 14:46:20.555886 12124 ts_tablet_manager.cc:1939] T 747b1af12d654817b6e45047037ab84f P 4dfda22ed5374cdaa1836ad0ab6f72ca: Deleting consensus metadata
I20260926 14:46:20.563019 11582 catalog_manager.cc:4841] TS 4dfda22ed5374cdaa1836ad0ab6f72ca (127.0.98.3:41383): tablet 747b1af12d654817b6e45047037ab84f (table test-table [id=4c439dcea0d64686a5ddebf691c5323c]) successfully deleted
W20260926 14:46:20.573530 12094 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:46:20.572828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34095 (local address 127.0.0.1:42606)
0926 14:46:20.573050 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 14:46:20.573065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:46:20.573161 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:46:20.573390 (+ 229us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34095: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 14:46:20.574325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 11952
I20260926 14:46:20.583082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 11565
I20260926 14:46:20.590492 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 14:46:20.935626 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 14:46:20.935750 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 14:46:21.270920 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 14:46:21.271 UTC [11151] LOG: received smart shutdown request
2026-09-26 14:46:21.274 UTC [11151] LOG: background worker "logical replication launcher" (PID 11160) exited with exit code 1
2026-09-26 14:46:21.274 UTC [11155] LOG: shutting down
2026-09-26 14:46:21.433 UTC [11151] LOG: database system is shut down
2026-09-26T14:46:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46496 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:46:21 dist-test-slave-s7k6 krb5kdc[12135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:46:21 dist-test-slave-s7k6 krb5kdc[12135](info): set up 2 sockets
Sep 26 14:46:21 dist-test-slave-s7k6 krb5kdc[12135](info): commencing operation
krb5kdc: starting...
W20260926 14:46:22.494673 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:46:22 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790433982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:46:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 14:46:22.661330 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskR2lxu1/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/postgres -l logfile start
2026-09-26 14:46:25.112 UTC [12165] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 14:46:25.112 UTC [12165] LOG: listening on IPv4 address "127.0.98.20", port 58927
2026-09-26 14:46:25.115 UTC [12165] LOG: listening on Unix socket "/tmp/.s.PGSQL.58927"
2026-09-26 14:46:25.142 UTC [12168] LOG: database system was shut down at 2026-09-26 14:46:23 UTC
2026-09-26 14:46:25.145 UTC [12165] LOG: database system is ready to accept connections
I20260926 14:46:26.018500 392 mini_postgres.cc:96] Postgres bound to 58927
2026-09-26 14:46:26.024 UTC [12176] FATAL: database "slave" does not exist
127.0.98.20:58927 - accepting connections
I20260926 14:46:26.025077 392 mini_ranger.cc:161] Starting Ranger...
I20260926 14:46:26.039455 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 14:46:26.498690 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 14:46:26.498777 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-admin
2026-09-26 14:46:26,760 [I] DB FLAVOR :POSTGRES
2026-09-26 14:46:26,761 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:46:26,761 [I] Checking connection..
2026-09-26 14:46:26,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 14:46:27,005 [I] Checking connection passed.
2026-09-26 14:46:27,006 [I] --------- Verifying version history table ---------
2026-09-26 14:46:27,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:46:27,254 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:46:27,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:46:27,519 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 14:46:27,519 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 14:46:27,519 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 14:46:27.762 UTC [12240] WARNING: there is no transaction in progress
2026-09-26 14:46:27,768 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 14:46:27,768 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 14:46:28,029 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 14:46:28,029 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:46:28,030 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 14:46:28,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 14:46:28,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s7k6.c.gcp-upstream.internal','N') ;"
2026-09-26 14:46:28,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:46:29,027 [I] Table x_portal_user does not exist in database ranger
2026-09-26 14:46:29,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:46:29,291 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 14:46:29,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:46:29,525 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:46:29,771 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 14:46:29,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 14:46:30.783 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:30.799 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:30.817 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:30.835 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:30.853 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:30.872 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:30.940 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:30.947 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:30.962 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:30.974 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:30.985 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:31.000 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:31.011 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:31.018 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:31.025 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:31.033 UTC [12348] WARNING: there is no transaction in progress
2026-09-26 14:46:31,680 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 14:46:31,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 14:46:31,900 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 14:46:31,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 14:46:32,148 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 14:46:32,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 14:46:32,408 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 14:46:32,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 14:46:32,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 14:46:32,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s7k6.c.gcp-upstream.internal';"
2026-09-26 14:46:33,121 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 14:46:33,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 14:46:33,354 [I] DB_PATCHES have already been applied
I20260926 14:46:33.359613 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskR2lxu1/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 14:46:33.359679 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 14:46:33.362915 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:47871
Sep 26, 2026 2:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 2:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 2:46:33 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 2:46:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 2:46:34 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 14:46:34.351343 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.326s user 0.000s sys 0.009s
Sep 26, 2026 2:46:55 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 2:46:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 2:46:55 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 2:46:55 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 2:46:56 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 14:46:56.692533 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 14:46:56 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434016, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 14:46:56.770677 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 14:46:56.779791 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 14:46:57.588059 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 14:46:57.588141 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-kms
I20260926 14:46:57.588158 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:58927
2026-09-26 14:46:57,746 [I] DB FLAVOR :POSTGRES
2026-09-26 14:46:57,746 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 14:46:57,746 [I] Checking connection
2026-09-26 14:46:57,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 14:46:57,977 [I] connection success
2026-09-26 14:46:57,978 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 14:46:57,978 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 14:46:57,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 14:46:58,220 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 14:46:58,221 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 14:46:58,221 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 14:46:58,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58927/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 14:46:58,480 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 14:46:58.954816 392 mini_ranger_kms.cc:310] Created kms service
I20260926 14:46:59.175892 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 14:46:59.222242 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 14:46:59 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434016, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:46:59.605401 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 14:46:59.605672 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskR2lxu1/build/debug/bin/postgresql.jar:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskR2lxu1/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 14:46:59.605703 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 14:46:59.609549 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12528
I20260926 14:46:59.609624 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:34235
Sep 26, 2026 2:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 2:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 2:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 2:47:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 2:47:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 2:47:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 14:47:01.662096 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.891s user 0.012s sys 0.130s
I20260926 14:47:01.662231 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 14:47:01.662314 392 mini_ranger_kms.cc:401] 127.0.98.20:34235/kms/v1/keys
Sep 26 14:47:01 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434021, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 14:47:02 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434021, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 14:47:02 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434021, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 14:47:04 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:47:04.894104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32927
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32927
--ranger_config_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34235/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:05.023788 12579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:05.024063 12579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:05.024101 12579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:05.028231 12579 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:47:05.028317 12579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:05.028337 12579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:05.028352 12579 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:47:05.028365 12579 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:47:05.034189 12579 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:34235/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32927
--ranger_config_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:05.035356 12579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:05.036437 12579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:05.046382 12585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:05.046394 12584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:05.046433 12579 server_base.cc:1034] running on GCE node
W20260926 14:47:05.046622 12587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:05.047493 12579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:05.048977 12579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:05.050154 12579 hybrid_clock.cc:648] HybridClock initialized: now 1790434025050124 us; error 40 us; skew 500 ppm
Sep 26 14:47:05 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:05.058346 12579 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:47:05.059605 12579 webserver.cc:460] Webserver started at http://127.0.98.62:43645/ using document root <none> and password file <none>
I20260926 14:47:05.060314 12579 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:05.060395 12579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:05.060626 12579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:47:05 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:47:05.109787 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "fb73f098381a4d629bf750646dedbffc"
format_stamp: "Formatted at 2026-09-26 14:47:05 on dist-test-slave-s7k6"
server_key: "26a5a05b6dd43b61f2721e7e770f93df"
server_key_iv: "130174042df9bfb901de93254566acc2"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:05.110450 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb73f098381a4d629bf750646dedbffc"
format_stamp: "Formatted at 2026-09-26 14:47:05 on dist-test-slave-s7k6"
server_key: "26a5a05b6dd43b61f2721e7e770f93df"
server_key_iv: "130174042df9bfb901de93254566acc2"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:05.114590 12579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:47:05.133643 12594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:05.135123 12579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:47:05.135262 12579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "fb73f098381a4d629bf750646dedbffc"
format_stamp: "Formatted at 2026-09-26 14:47:05 on dist-test-slave-s7k6"
server_key: "26a5a05b6dd43b61f2721e7e770f93df"
server_key_iv: "130174042df9bfb901de93254566acc2"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:05.135393 12579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:05.154906 12579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:05.157981 12579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:05.158164 12579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:05.166002 12579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32927
I20260926 14:47:05.166018 12646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32927 every 8 connection(s)
I20260926 14:47:05.167054 12579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:47:05.170552 12647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:05.172130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 12579
I20260926 14:47:05.172245 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:47:05.172521 392 external_mini_cluster.cc:1428] Setting key 0c8f8a7147fe114bd85834545d25b9f5
I20260926 14:47:05.177778 12647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc: Bootstrap starting.
I20260926 14:47:05.180562 12647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc: Neither blocks nor log segments found. Creating new log.
Sep 26 14:47:05 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:05.181448 12647 log.cc:824] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:05.183923 12647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc: No bootstrap required, opened a new log
I20260926 14:47:05.186513 12650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:05.173923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.0.98.62:32927)
0926 14:47:05.174315 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:05.174331 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:05.174366 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:47:05.175055 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:05.175073 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:05.175469 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:05.175781 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:05.176579 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:05.177546 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:05.178440 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:05.178972 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:05.179119 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:05.181591 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:05.181597 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:05.181616 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:05.181647 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:05.183755 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:05.184232 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:05.184238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:05.184243 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:05.184355 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:05.184612 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:05.184616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:05.184617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:05.184997 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:05.185167 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:05.185477 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 14:47:05.185764 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260926 14:47:05.187958 12647 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 14:47:05.188244 12647 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:05.188350 12647 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb73f098381a4d629bf750646dedbffc, State: Initialized, Role: FOLLOWER
I20260926 14:47:05.188855 12647 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 14:47:05.189018 12647 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:47:05.189081 12647 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:47:05.189175 12647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:05.190374 12647 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 14:47:05.190806 12647 leader_election.cc:302] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb73f098381a4d629bf750646dedbffc; no voters:
I20260926 14:47:05.191229 12647 leader_election.cc:288] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:47:05.191465 12652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:47:05.191886 12652 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 1 LEADER]: Becoming Leader. State: Replica: fb73f098381a4d629bf750646dedbffc, State: Running, Role: LEADER
I20260926 14:47:05.192404 12652 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 14:47:05.192675 12647 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:47:05.193902 12653 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb73f098381a4d629bf750646dedbffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } } }
I20260926 14:47:05.194043 12653 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:05.194242 12653 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb73f098381a4d629bf750646dedbffc. Latest consensus state: current_term: 1 leader_uuid: "fb73f098381a4d629bf750646dedbffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } } }
I20260926 14:47:05.194311 12653 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:05.195223 12647 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 14:47:05.195860 12661 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:47:05.201220 12661 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:47:05.212709 12661 catalog_manager.cc:1347] Generated new cluster ID: af99464d358e4642ae5befe9e1944a0c
I20260926 14:47:05.212781 12661 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:47:05.228850 12661 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:47:05.230366 12661 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:47:05.240713 12661 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc: Generated new TSK 0
I20260926 14:47:05.241669 12661 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:47:05.962818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:06.129263 12692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:06.129674 12692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:06.129750 12692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:06.135993 12692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:06.136076 12692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:06.136162 12692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:47:06.144490 12692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:34235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:06.146195 12692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:06.147766 12692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:06.161801 12698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:06.162057 12699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:06.162395 12701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:06.163748 12692 server_base.cc:1034] running on GCE node
I20260926 14:47:06.164299 12692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:06.165184 12692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:06.166379 12692 hybrid_clock.cc:648] HybridClock initialized: now 1790434026166331 us; error 73 us; skew 500 ppm
Sep 26 14:47:06 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:06.169453 12692 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:47:06.170790 12692 webserver.cc:460] Webserver started at http://127.0.98.1:38285/ using document root <none> and password file <none>
I20260926 14:47:06.171620 12692 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:06.171773 12692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:06.172057 12692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:47:06 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:47:06.192401 12692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "5964cf9e62fb491e9ac8416c9c01e6eb"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "1cc2e858451538d717f76184c3729d7e"
server_key_iv: "1628f7d3c341f0d2d3b6ca8f7e84c454"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:06.193234 12692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5964cf9e62fb491e9ac8416c9c01e6eb"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "1cc2e858451538d717f76184c3729d7e"
server_key_iv: "1628f7d3c341f0d2d3b6ca8f7e84c454"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:06.199424 12692 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260926 14:47:06.218425 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:06.220412 12692 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:47:06.220569 12692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "5964cf9e62fb491e9ac8416c9c01e6eb"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "1cc2e858451538d717f76184c3729d7e"
server_key_iv: "1628f7d3c341f0d2d3b6ca8f7e84c454"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:06.220723 12692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:06.256431 12692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:06.260149 12692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:06.260339 12692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:06.260938 12692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:06.261945 12692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:06.262001 12692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:06.262092 12692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:06.262125 12692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:06.272315 12692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43701
I20260926 14:47:06.272545 12824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43701 every 8 connection(s)
I20260926 14:47:06.273378 12692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:47:06.281790 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 12692
I20260926 14:47:06.281934 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:47:06.282243 392 external_mini_cluster.cc:1428] Setting key 36e8c2726f3f12fd3ddd4baee958b754
Sep 26 14:47:06 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:06.291671 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:06.275486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54435 (local address 127.0.98.62:32927)
0926 14:47:06.275748 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:06.275757 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:06.276690 (+ 933us) server_negotiation.cc:407] Connection header received
0926 14:47:06.278632 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:06.278637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:06.278698 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:06.278806 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:06.280563 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:06.281448 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:06.283255 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:06.283637 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:06.283720 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:06.286982 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:06.286987 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:06.286990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:06.287020 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:06.288931 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:06.289647 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:06.289651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:06.289653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:06.289748 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:06.290170 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:06.290174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:06.290176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:06.290422 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:06.290505 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:06.291321 (+ 816us) server_negotiation.cc:299] Negotiation successful
0926 14:47:06.291468 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":82,"threads_started":1}
I20260926 14:47:06.293619 12828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:06.276018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32927 (local address 127.0.98.1:54435)
0926 14:47:06.276524 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:06.276581 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:06.278359 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:06.279269 (+ 910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:06.279278 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:06.279648 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:06.280366 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:06.280387 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:06.281629 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:06.281632 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:06.283091 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:06.283109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:06.283951 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:06.283955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:06.284183 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:06.284902 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:06.284925 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:06.286698 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:06.289107 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:06.289115 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:06.289138 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:06.289524 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:06.289853 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:06.289868 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:06.289870 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:06.290063 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:06.290574 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:06.290580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:06.290997 (+ 417us) client_negotiation.cc:769] Sending connection context
0926 14:47:06.292159 (+ 1162us) client_negotiation.cc:240] Negotiation successful
0926 14:47:06.292534 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":122,"threads_started":1}
I20260926 14:47:06.295356 12825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32927
I20260926 14:47:06.295608 12825 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:06.296195 12825 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
I20260926 14:47:06.298038 12611 ts_manager.cc:194] Registered new tserver with Master: 5964cf9e62fb491e9ac8416c9c01e6eb (127.0.98.1:43701)
I20260926 14:47:06.299710 12611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54435
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:47:06.329744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:47:06 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 14:47:06.474325 12831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:06.474807 12831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:06.474943 12831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:06.479856 12831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:06.479934 12831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:06.480017 12831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:47:06.485785 12831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:34235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:06.487138 12831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:06.488533 12831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:06.498978 12838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:06.499358 12840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:06.499852 12837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:06.500543 12831 server_base.cc:1034] running on GCE node
I20260926 14:47:06.500950 12831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:06.501543 12831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:06.502725 12831 hybrid_clock.cc:648] HybridClock initialized: now 1790434026502694 us; error 44 us; skew 500 ppm
Sep 26 14:47:06 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:06.506501 12831 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:47:06.507745 12831 webserver.cc:460] Webserver started at http://127.0.98.2:41843/ using document root <none> and password file <none>
I20260926 14:47:06.508569 12831 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:06.508803 12831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:06.509130 12831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:47:06 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:47:06.529781 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "d423e13daa94366e95c0e82a73557d20"
server_key_iv: "b56b9e8703536ee3b8408dee0120947e"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:06.530380 12831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "d423e13daa94366e95c0e82a73557d20"
server_key_iv: "b56b9e8703536ee3b8408dee0120947e"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:06.534454 12831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:47:06.550988 12847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:06.553053 12831 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:47:06.553282 12831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "d423e13daa94366e95c0e82a73557d20"
server_key_iv: "b56b9e8703536ee3b8408dee0120947e"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:06.553630 12831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:06.573341 12831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:06.576938 12831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:06.577170 12831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:06.577909 12831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:06.578996 12831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:06.579101 12831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:06.579200 12831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:06.579267 12831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:06.589946 12831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37619
I20260926 14:47:06.591285 12831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:47:06.592804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 12831
I20260926 14:47:06.592948 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:47:06.593266 392 external_mini_cluster.cc:1428] Setting key fe09cb1780be1c44bfeac200597f570a
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 14:47:06.604712 12960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37619 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 26 14:47:06 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:06.629773 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:06.608163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45023 (local address 127.0.98.62:32927)
0926 14:47:06.608307 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:06.608311 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:06.613910 (+ 5599us) server_negotiation.cc:407] Connection header received
0926 14:47:06.615380 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:06.615384 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:06.615460 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:06.615518 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:06.617333 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:06.618147 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:06.619877 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:06.620241 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:06.620326 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:06.624806 (+ 4480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:06.624811 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:06.624814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:06.624848 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:06.626811 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:06.627514 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:06.627519 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:06.627521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:06.627626 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:06.628110 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:06.628114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:06.628116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:06.628430 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:06.628530 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:06.629418 (+ 888us) server_negotiation.cc:299] Negotiation successful
0926 14:47:06.629586 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:47:06.630995 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:06.631978 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:06.607012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32927 (local address 127.0.98.2:45023)
0926 14:47:06.613703 (+ 6691us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:06.613779 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:06.615073 (+ 1294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:06.616016 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:06.616026 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:06.616429 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:06.617158 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:06.617177 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:06.618300 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:06.618303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:06.619739 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:06.619751 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:06.620572 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:06.620576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:06.620778 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:06.621772 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:06.621794 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:06.624591 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:06.626938 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:06.626973 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:06.626993 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:06.627385 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:06.627724 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:06.627727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:06.627729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:06.627961 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:06.628590 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:06.628598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:06.629048 (+ 450us) client_negotiation.cc:769] Sending connection context
0926 14:47:06.630289 (+ 1241us) client_negotiation.cc:240] Negotiation successful
0926 14:47:06.630720 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6395,"thread_start_us":152,"threads_started":1}
I20260926 14:47:06.633630 12961 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32927
I20260926 14:47:06.633917 12961 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:06.634563 12961 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
I20260926 14:47:06.635919 12611 ts_manager.cc:194] Registered new tserver with Master: 6c7bc8acbbe54bc08f9c21180ab24ef6 (127.0.98.2:37619)
I20260926 14:47:06.636739 12611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45023
W20260926 14:47:06.806401 12966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:06.807265 12966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:06.807420 12966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:06.813838 12966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:06.813984 12966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:06.814122 12966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:47:06.822556 12966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:34235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:06.824293 12966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:06.825925 12966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:06.853040 12974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:06.853561 12972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:06.854789 12971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:06.855856 12966 server_base.cc:1034] running on GCE node
I20260926 14:47:06.856586 12966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:06.857441 12966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 26 14:47:06 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:47:06.860716 12966 hybrid_clock.cc:648] HybridClock initialized: now 1790434026860626 us; error 91 us; skew 500 ppm
Sep 26 14:47:06 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:06.864373 12966 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:47:06.865536 12966 webserver.cc:460] Webserver started at http://127.0.98.3:42747/ using document root <none> and password file <none>
I20260926 14:47:06.866241 12966 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:06.866317 12966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:06.866546 12966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 14:47:06 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:47:06.891801 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "accf1f4b0515453bbfb6ca19eb308255"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "0b6f2a9057f0dd271ff0d4375ad697aa"
server_key_iv: "1059347bcaf04e237a545dd85e5e027d"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:06.892554 12966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "accf1f4b0515453bbfb6ca19eb308255"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "0b6f2a9057f0dd271ff0d4375ad697aa"
server_key_iv: "1059347bcaf04e237a545dd85e5e027d"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:06.898799 12966 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 14:47:06.921411 12981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:06.923220 12966 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 14:47:06.923369 12966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "accf1f4b0515453bbfb6ca19eb308255"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "0b6f2a9057f0dd271ff0d4375ad697aa"
server_key_iv: "1059347bcaf04e237a545dd85e5e027d"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:06.923527 12966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:06.963368 12966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:06.974560 12966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:06.974848 12966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:06.975565 12966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:06.976667 12966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:06.976760 12966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:06.976841 12966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:06.976887 12966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:06.986543 12966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37015
I20260926 14:47:06.987768 12966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:47:06.992350 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 12966
I20260926 14:47:06.992487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:47:06.992816 392 external_mini_cluster.cc:1428] Setting key 214500ba7ddaf70d35dafe1d70fcbd80
I20260926 14:47:06.993705 13094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37015 every 8 connection(s)
Sep 26 14:47:07 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:07.010342 12827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:06.996328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35857 (local address 127.0.98.62:32927)
0926 14:47:06.996496 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:06.996500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:06.996964 (+ 464us) server_negotiation.cc:407] Connection header received
0926 14:47:06.998245 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:06.998248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:06.998301 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:06.998349 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:07.000043 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.000821 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.002529 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.002854 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.002926 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:07.006018 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:07.006022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:07.006024 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:07.006053 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:07.007663 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:07.008267 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:07.008271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:07.008272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:07.008356 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:07.008981 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:07.008985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:07.008987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:07.009221 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:07.009296 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:07.010045 (+ 749us) server_negotiation.cc:299] Negotiation successful
0926 14:47:07.010179 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:47:07.012272 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:06.995037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32927 (local address 127.0.98.3:35857)
0926 14:47:06.996794 (+ 1757us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:06.996863 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:06.997995 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:06.998823 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:06.998832 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:06.999197 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:06.999875 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:06.999895 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.000984 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.000987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:07.002385 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:07.002396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.003130 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.003133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:07.003324 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:07.004034 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:07.004055 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:07.005796 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:07.007800 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:07.007805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:07.007823 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:07.008146 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:07.008554 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:07.008558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:07.008560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:07.008873 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:07.009388 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:07.009395 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:07.009766 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 14:47:07.010559 (+ 793us) client_negotiation.cc:240] Negotiation successful
0926 14:47:07.011225 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1510,"thread_start_us":252,"threads_started":1}
I20260926 14:47:07.013512 13095 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32927
I20260926 14:47:07.013762 13095 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:07.014328 13095 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
I20260926 14:47:07.015370 12611 ts_manager.cc:194] Registered new tserver with Master: accf1f4b0515453bbfb6ca19eb308255 (127.0.98.3:37015)
I20260926 14:47:07.016074 12611 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35857
I20260926 14:47:07.018573 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:47:07.019471 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 12692
I20260926 14:47:07.026845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 12831
I20260926 14:47:07.032629 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 12966
I20260926 14:47:07.037479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 12579
I20260926 14:47:07.043413 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32927
--webserver_interface=127.0.98.62
--webserver_port=43645
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32927
--ranger_config_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34235/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34235/kms with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:07.155530 13098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:07.155827 13098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:07.155871 13098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:07.160105 13098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:47:07.160171 13098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:07.160189 13098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:07.160204 13098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:47:07.160218 13098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:47:07.166035 13098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34235/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32927
--ranger_config_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:07.167373 13098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:07.168454 13098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:07.177390 13103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:07.177372 13104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:07.177376 13106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:07.177771 13098 server_base.cc:1034] running on GCE node
I20260926 14:47:07.178423 13098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:07.179849 13098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:07.181038 13098 hybrid_clock.cc:648] HybridClock initialized: now 1790434027181021 us; error 29 us; skew 500 ppm
Sep 26 14:47:07 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:07.184440 13098 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:47:07.185555 13098 webserver.cc:460] Webserver started at http://127.0.98.62:43645/ using document root <none> and password file <none>
I20260926 14:47:07.186239 13098 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:07.186334 13098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:47:07 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:47:07.248155 13098 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:47:07.250463 13114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:07.251941 13098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:47:07.252068 13098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "fb73f098381a4d629bf750646dedbffc"
format_stamp: "Formatted at 2026-09-26 14:47:05 on dist-test-slave-s7k6"
server_key: "26a5a05b6dd43b61f2721e7e770f93df"
server_key_iv: "130174042df9bfb901de93254566acc2"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:07.252642 13098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:07.269404 13098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:07.272336 13098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:07.272526 13098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:07.279742 13098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32927
I20260926 14:47:07.279771 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32927 every 8 connection(s)
I20260926 14:47:07.280893 13098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:47:07.285331 13167 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:47:07.287076 13167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc: Bootstrap starting.
I20260926 14:47:07.290874 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 13098
I20260926 14:47:07.291352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43701
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32927
--webserver_port=38285
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34235/kms with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:07.303768 13167 log.cc:824] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:07.311105 13167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:47:07.311506 13167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc: Bootstrap complete.
I20260926 14:47:07.315549 13167 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 14:47:07.315948 13167 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb73f098381a4d629bf750646dedbffc, State: Initialized, Role: FOLLOWER
I20260926 14:47:07.316432 13167 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 14:47:07.316565 13167 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:47:07.316625 13167 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:47:07.316720 13167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:47:07.318591 13167 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 14:47:07.318928 13167 leader_election.cc:302] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb73f098381a4d629bf750646dedbffc; no voters:
I20260926 14:47:07.319238 13167 leader_election.cc:288] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:47:07.319332 13171 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:47:07.319543 13171 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [term 2 LEADER]: Becoming Leader. State: Replica: fb73f098381a4d629bf750646dedbffc, State: Running, Role: LEADER
I20260926 14:47:07.319839 13171 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } }
I20260926 14:47:07.320464 13167 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:47:07.320788 13171 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb73f098381a4d629bf750646dedbffc. Latest consensus state: current_term: 2 leader_uuid: "fb73f098381a4d629bf750646dedbffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } } }
I20260926 14:47:07.320907 13171 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:07.321260 13175 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:47:07.321441 13172 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fb73f098381a4d629bf750646dedbffc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb73f098381a4d629bf750646dedbffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32927 } } }
I20260926 14:47:07.321620 13172 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:07.325310 13175 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:47:07.325891 13167 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 14:47:07.325951 13175 catalog_manager.cc:1259] Loaded cluster ID: af99464d358e4642ae5befe9e1944a0c
I20260926 14:47:07.326231 13175 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:47:07.328634 13175 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:47:07.329291 13175 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc: Loaded TSK: 0
I20260926 14:47:07.330482 13175 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:47:07.432994 13169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:07.433373 13169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:07.433444 13169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:07.439723 13169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:07.439811 13169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:07.439898 13169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:47:07.447588 13169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43701
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38285
--tserver_master_addrs=127.0.98.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:07.448864 13169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:07.450176 13169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:07.466265 13204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:07.466490 13207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:07.467132 13169 server_base.cc:1034] running on GCE node
W20260926 14:47:07.467342 13205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:07.467868 13169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:07.468696 13169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:07.469897 13169 hybrid_clock.cc:648] HybridClock initialized: now 1790434027469856 us; error 51 us; skew 500 ppm
Sep 26 14:47:07 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:07.473789 13169 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:47:07.475075 13169 webserver.cc:460] Webserver started at http://127.0.98.1:38285/ using document root <none> and password file <none>
I20260926 14:47:07.475833 13169 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:07.475950 13169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:47:07 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:47:07.515827 13169 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.003s sys 0.000s
I20260926 14:47:07.523064 13215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:07.524744 13169 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260926 14:47:07.524880 13169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "5964cf9e62fb491e9ac8416c9c01e6eb"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "1cc2e858451538d717f76184c3729d7e"
server_key_iv: "1628f7d3c341f0d2d3b6ca8f7e84c454"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:07.525681 13169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:07.542841 13169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:07.545946 13169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:07.546106 13169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:07.546816 13169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:07.547792 13169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:07.547850 13169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:07.547899 13169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:07.547916 13169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:07.561219 13169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43701
I20260926 14:47:07.562597 13169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:47:07.563041 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 13169
I20260926 14:47:07.563504 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37619
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32927
--webserver_port=41843
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34235/kms with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:07.566665 13328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43701 every 8 connection(s)
Sep 26 14:47:07 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:07.591225 13332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:07.569216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32927 (local address 127.0.98.1:54617)
0926 14:47:07.571529 (+ 2313us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:07.571589 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:07.572820 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:07.574385 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:07.574395 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:07.574789 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:07.575512 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:07.575547 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.577889 (+ 2342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.577893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:07.579299 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:07.579312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.580417 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.580422 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:07.580627 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:07.581473 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:07.581496 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:07.584885 (+ 3389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:07.587574 (+ 2689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:07.587581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:07.587598 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:07.587961 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:07.588407 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:07.588410 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:07.588412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:07.588592 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:07.589336 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:07.589342 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:07.589663 (+ 321us) client_negotiation.cc:769] Sending connection context
0926 14:47:07.589845 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 14:47:07.590188 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2071,"thread_start_us":162,"threads_started":1}
I20260926 14:47:07.593369 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:07.570783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54617 (local address 127.0.98.62:32927)
0926 14:47:07.571407 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:07.571429 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:07.571739 (+ 310us) server_negotiation.cc:407] Connection header received
0926 14:47:07.573381 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:07.573398 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:07.573874 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:07.575780 (+ 1906us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:07.576096 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.577727 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.579451 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.580233 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.580400 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:07.585179 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:07.585187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:07.585209 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:07.585246 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:07.587336 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:07.588133 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:07.588140 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:07.588147 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:07.588297 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:07.588701 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:07.588704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:07.588706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:07.589204 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:07.591952 (+ 2748us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:07.592134 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 14:47:07.592454 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":123,"threads_started":1}
I20260926 14:47:07.596242 13329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32927
I20260926 14:47:07.596529 13329 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:07.597157 13329 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
W20260926 14:47:07.599647 13329 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32927 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 14:47:07.599779 13329 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:07.599965 13329 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
I20260926 14:47:07.600867 13329 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:07.601043 13329 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
W20260926 14:47:07.601827 13329 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:47:07.703114 13331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:07.703383 13331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:07.703444 13331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:07.708942 13331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:07.709035 13331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:07.709120 13331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:47:07.717108 13331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41843
--tserver_master_addrs=127.0.98.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:07.718566 13331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:07.720119 13331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:07.732405 13341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:07.732515 13339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:07.732430 13338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:07.732973 13331 server_base.cc:1034] running on GCE node
I20260926 14:47:07.733513 13331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:07.734256 13331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:07.735411 13331 hybrid_clock.cc:648] HybridClock initialized: now 1790434027735367 us; error 59 us; skew 500 ppm
Sep 26 14:47:07 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:07.738922 13331 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:47:07.739959 13331 webserver.cc:460] Webserver started at http://127.0.98.2:41843/ using document root <none> and password file <none>
I20260926 14:47:07.740654 13331 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:07.740742 13331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:47:07 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:47:07.767448 13331 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 14:47:07.770560 13349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:07.772230 13331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:47:07.772359 13331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "d423e13daa94366e95c0e82a73557d20"
server_key_iv: "b56b9e8703536ee3b8408dee0120947e"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:07.773098 13331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:07.788669 13331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:07.792284 13331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:07.792508 13331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:07.793288 13331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:07.794412 13331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:07.794479 13331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:07.794539 13331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:07.794564 13331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:07.805114 13331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37619
I20260926 14:47:07.805406 13463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37619 every 8 connection(s)
I20260926 14:47:07.806219 13331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:47:07.812409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 13331
I20260926 14:47:07.812891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:34235/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37015
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32927
--webserver_port=42747
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34235/kms with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:47:07 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:07.822177 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:07.808178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46537 (local address 127.0.98.62:32927)
0926 14:47:07.808330 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:07.808335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:07.809229 (+ 894us) server_negotiation.cc:407] Connection header received
0926 14:47:07.810173 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:07.810177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:07.810235 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:07.810339 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:07.811477 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.812337 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.813955 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.814311 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.814436 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:07.816923 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:07.816926 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:07.816929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:07.816954 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:07.819907 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:07.820491 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:07.820495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:07.820497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:07.820577 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:07.820929 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:07.820933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:07.820935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:07.821145 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:07.821289 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:07.821877 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 14:47:07.822006 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:47:07.823426 13466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:07.808546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32927 (local address 127.0.98.2:46537)
0926 14:47:07.809043 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:07.809096 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:07.809956 (+ 860us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:07.810488 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:07.810494 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:07.810797 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:07.811324 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:07.811339 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.812483 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.812487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:07.813813 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:07.813825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:07.814483 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:07.814487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:07.814684 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:07.815429 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:07.815446 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:07.816735 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:07.820042 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:07.820048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:07.820065 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:07.820400 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:07.820657 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:07.820661 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:07.820663 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:07.820845 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:07.821365 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:07.821374 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:07.821641 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 14:47:07.821819 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 14:47:07.822586 (+ 767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":116,"threads_started":1}
I20260926 14:47:07.824596 13464 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32927
I20260926 14:47:07.824843 13464 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:07.825338 13464 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
W20260926 14:47:07.826882 13464 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32927 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 14:47:07.827044 13464 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:07.827244 13464 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
I20260926 14:47:07.828171 13464 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:07.828349 13464 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
W20260926 14:47:07.829126 13464 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:47:07.945724 13467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:07.946108 13467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:07.946177 13467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:07.952498 13467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:07.952623 13467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:07.952728 13467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:47:07.961053 13467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:34235/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42747
--tserver_master_addrs=127.0.98.62:32927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:07.962563 13467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:07.963972 13467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:07.974426 13472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:07.974809 13475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:07.974912 13473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:07.975212 13467 server_base.cc:1034] running on GCE node
I20260926 14:47:07.975616 13467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:07.976261 13467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:07.977537 13467 hybrid_clock.cc:648] HybridClock initialized: now 1790434027977506 us; error 51 us; skew 500 ppm
Sep 26 14:47:07 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:07.981276 13467 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:47:07.982470 13467 webserver.cc:460] Webserver started at http://127.0.98.3:42747/ using document root <none> and password file <none>
I20260926 14:47:07.983256 13467 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:07.983367 13467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 14:47:07 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:47:08.004657 13467 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260926 14:47:08.007788 13483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:08.009447 13467 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:47:08.009572 13467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "accf1f4b0515453bbfb6ca19eb308255"
format_stamp: "Formatted at 2026-09-26 14:47:06 on dist-test-slave-s7k6"
server_key: "0b6f2a9057f0dd271ff0d4375ad697aa"
server_key_iv: "1059347bcaf04e237a545dd85e5e027d"
server_key_version: "kuduclusterkey@0"
I20260926 14:47:08.010340 13467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:08.054854 13467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:08.058696 13467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:08.058969 13467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:08.059779 13467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:08.060949 13467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:08.061014 13467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:08.061164 13467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:08.061211 13467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:08.072424 13467 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37015
I20260926 14:47:08.072440 13603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37015 every 8 connection(s)
I20260926 14:47:08.073786 13467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:47:08.081650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 13467
Sep 26 14:47:08 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:08.092510 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:08.082737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.0.98.62:32927)
0926 14:47:08.083045 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:08.083052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:08.083066 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:47:08.083103 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:08.083105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:08.083144 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:08.083244 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:08.084262 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.085039 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.086700 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.087057 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.087152 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:08.088644 (+ 1492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:08.088648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:08.088651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:08.088685 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:08.090501 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:08.091020 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:08.091024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:08.091026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:08.091108 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:08.091402 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:08.091406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:08.091408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:08.091644 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:08.091849 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:08.092173 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 14:47:08.092313 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":117,"threads_started":1}
I20260926 14:47:08.092902 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:08.079578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58587 (local address 127.0.98.62:32927)
0926 14:47:08.079724 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:08.079729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:08.080960 (+ 1231us) server_negotiation.cc:407] Connection header received
0926 14:47:08.081946 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:08.081949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:08.082011 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:08.082083 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:08.083678 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.084190 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.085617 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.085920 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.086033 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:08.088974 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:08.088977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:08.088980 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:08.089024 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:08.090595 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:08.091255 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:08.091259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:08.091260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:08.091329 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:08.091687 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:08.091690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:08.091692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:08.091885 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:08.091966 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:08.092653 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 14:47:08.092769 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:47:08.093770 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:08.080195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32927 (local address 127.0.98.3:58587)
0926 14:47:08.080777 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:08.080823 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:08.081701 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:08.082418 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:08.082428 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:08.082831 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:08.083531 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:08.083548 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.084327 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.084331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:08.085502 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:08.085511 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.086209 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.086213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:08.086405 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:08.087068 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:08.087092 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:08.088780 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:08.090707 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:08.090714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:08.090734 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:08.091110 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:08.091430 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:08.091433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:08.091435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:08.091584 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:08.092019 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:08.092026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:08.092400 (+ 374us) client_negotiation.cc:769] Sending connection context
0926 14:47:08.092579 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 14:47:08.092912 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":183,"threads_started":1}
I20260926 14:47:08.095162 13604 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32927
I20260926 14:47:08.095407 13604 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:08.095942 13604 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
I20260926 14:47:08.097301 13131 ts_manager.cc:194] Registered new tserver with Master: accf1f4b0515453bbfb6ca19eb308255 (127.0.98.3:37015)
I20260926 14:47:08.098495 13131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58587
Sep 26 14:47:08 dist-test-slave-s7k6 krb5kdc[12135](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:08.602056 13329 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:08.602363 13329 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
I20260926 14:47:08.603200 13131 ts_manager.cc:194] Registered new tserver with Master: 5964cf9e62fb491e9ac8416c9c01e6eb (127.0.98.1:43701)
I20260926 14:47:08.604044 13131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54617
Sep 26 14:47:08 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 14:47:08.829403 13464 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:08.829721 13464 heartbeater.cc:511] Master 127.0.98.62:32927 requested a full tablet report, sending...
I20260926 14:47:08.830582 13130 ts_manager.cc:194] Registered new tserver with Master: 6c7bc8acbbe54bc08f9c21180ab24ef6 (127.0.98.2:37619)
I20260926 14:47:08.831435 13130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:46537
I20260926 14:47:08.841341 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:47:08.859304 13619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:08.843054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44594 (local address 127.0.98.62:32927)
0926 14:47:08.843306 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:08.843311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:08.847056 (+ 3745us) server_negotiation.cc:407] Connection header received
0926 14:47:08.850054 (+ 2998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:08.850058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:08.850124 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:08.850206 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:08.851258 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.852111 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.853297 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.853647 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.853742 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:08.855532 (+ 1790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:08.855537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:08.855540 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:08.855570 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:08.857443 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:08.857942 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:08.857946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:08.857948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:08.858045 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:08.858314 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:08.858317 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:08.858319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:08.858553 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:08.858690 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:08.858972 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 14:47:08.859116 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260926 14:47:08.863757 13130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:47:08.868417 13130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:47:08.892707 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:08.884679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43701 (local address 127.0.0.1:52894)
0926 14:47:08.885815 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:08.885826 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:08.885903 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:08.887255 (+ 1352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:08.887258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:08.887294 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:08.887383 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:08.887389 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.888839 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.888842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:08.890256 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:08.890267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.891590 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.891594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:08.892349 (+ 755us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:08.892373 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:47:08.892431 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 14:47:08.892554 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"mutex_wait_us":631,"thread_start_us":83,"threads_started":1}
I20260926 14:47:08.893020 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:08.883838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.0.98.2:37619)
0926 14:47:08.885113 (+ 1275us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:08.885122 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:08.885852 (+ 730us) server_negotiation.cc:407] Connection header received
0926 14:47:08.885935 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:08.885940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:08.886168 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:08.886311 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:08.886844 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.888920 (+ 2076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.890449 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.891310 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.891406 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:08.891538 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:08.892781 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0926 14:47:08.892886 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":116,"threads_started":1}
I20260926 14:47:08.893464 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:08.883042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37619 (local address 127.0.0.1:43550)
0926 14:47:08.885651 (+ 2609us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:08.885664 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:08.885761 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:08.886497 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:08.886503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:08.886525 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:08.886657 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:08.886663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.889041 (+ 2378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.889044 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:08.890317 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:08.890329 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.891596 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.891599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:08.891745 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:08.891784 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 14:47:08.892578 (+ 794us) client_negotiation.cc:240] Negotiation successful
0926 14:47:08.892657 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2546,"mutex_wait_us":15,"thread_start_us":97,"threads_started":1}
I20260926 14:47:08.894627 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:08.884810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.0.98.1:43701)
0926 14:47:08.886573 (+ 1763us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:08.886580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:08.886605 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:47:08.886681 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:08.886686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:08.886928 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:08.887069 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:08.887591 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.888709 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.890410 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.891263 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.891375 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:08.891531 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:08.894391 (+ 2860us) server_negotiation.cc:299] Negotiation successful
0926 14:47:08.894496 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1224,"thread_start_us":62,"threads_started":1}
I20260926 14:47:08.897179 13263 tablet_service.cc:1467] Processing CreateTablet for tablet 220a83294c7b46ba9a874e4ca2626e9a (DEFAULT_TABLE table=test-table [id=adfe46ff763e493cbb53cff224d25421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:08.897131 13398 tablet_service.cc:1467] Processing CreateTablet for tablet 220a83294c7b46ba9a874e4ca2626e9a (DEFAULT_TABLE table=test-table [id=adfe46ff763e493cbb53cff224d25421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:08.898275 13263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220a83294c7b46ba9a874e4ca2626e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:08.898275 13398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220a83294c7b46ba9a874e4ca2626e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:08.899729 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:08.882941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37015 (local address 127.0.0.1:56814)
0926 14:47:08.885392 (+ 2451us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:08.885450 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:08.885585 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:08.895500 (+ 9915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:08.895503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:08.895523 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:08.895610 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:08.895615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.897198 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.897202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:08.898304 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:08.898315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.899380 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.899384 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:08.899470 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:08.899496 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:47:08.899539 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 14:47:08.899602 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2257,"thread_start_us":116,"threads_started":1}
I20260926 14:47:08.900413 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:08.883936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56814 (local address 127.0.98.3:37015)
0926 14:47:08.891860 (+ 7924us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:08.891867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:08.891892 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:47:08.891968 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:08.891974 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:08.892180 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:08.895763 (+ 3583us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:08.895912 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.897003 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.898456 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:08.899241 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:08.899986 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:08.900133 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:08.900180 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:47:08.900268 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7808,"thread_start_us":67,"threads_started":1}
I20260926 14:47:08.903538 13538 tablet_service.cc:1467] Processing CreateTablet for tablet 220a83294c7b46ba9a874e4ca2626e9a (DEFAULT_TABLE table=test-table [id=adfe46ff763e493cbb53cff224d25421]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:08.904755 13538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220a83294c7b46ba9a874e4ca2626e9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:08.910879 13630 tablet_bootstrap.cc:492] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb: Bootstrap starting.
I20260926 14:47:08.914502 13630 tablet_bootstrap.cc:654] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:08.914484 13632 tablet_bootstrap.cc:492] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: Bootstrap starting.
I20260926 14:47:08.915050 13631 tablet_bootstrap.cc:492] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: Bootstrap starting.
I20260926 14:47:08.916981 13632 tablet_bootstrap.cc:654] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:08.917287 13630 log.cc:824] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:08.917855 13632 log.cc:824] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:08.919039 13631 tablet_bootstrap.cc:654] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:08.920125 13632 tablet_bootstrap.cc:492] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: No bootstrap required, opened a new log
I20260926 14:47:08.920331 13632 ts_tablet_manager.cc:1397] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:47:08.921185 13631 log.cc:824] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:08.923503 13630 tablet_bootstrap.cc:492] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb: No bootstrap required, opened a new log
I20260926 14:47:08.923799 13630 ts_tablet_manager.cc:1397] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260926 14:47:08.925238 13632 raft_consensus.cc:348] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accf1f4b0515453bbfb6ca19eb308255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37015 } } peers { permanent_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37619 } } peers { permanent_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43701 } }
I20260926 14:47:08.925558 13632 raft_consensus.cc:374] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:08.925614 13632 raft_consensus.cc:729] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: accf1f4b0515453bbfb6ca19eb308255, State: Initialized, Role: FOLLOWER
I20260926 14:47:08.926177 13632 consensus_queue.cc:260] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accf1f4b0515453bbfb6ca19eb308255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37015 } } peers { permanent_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37619 } } peers { permanent_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43701 } }
I20260926 14:47:08.927337 13632 ts_tablet_manager.cc:1428] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
I20260926 14:47:08.927722 13604 heartbeater.cc:503] Master 127.0.98.62:32927 was elected leader, sending a full tablet report...
I20260926 14:47:08.928282 13630 raft_consensus.cc:348] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accf1f4b0515453bbfb6ca19eb308255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37015 } } peers { permanent_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37619 } } peers { permanent_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43701 } }
I20260926 14:47:08.928581 13630 raft_consensus.cc:374] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:08.928663 13630 raft_consensus.cc:729] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5964cf9e62fb491e9ac8416c9c01e6eb, State: Initialized, Role: FOLLOWER
I20260926 14:47:08.929404 13630 consensus_queue.cc:260] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accf1f4b0515453bbfb6ca19eb308255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37015 } } peers { permanent_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37619 } } peers { permanent_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43701 } }
I20260926 14:47:08.930858 13630 ts_tablet_manager.cc:1428] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 14:47:08.931258 13329 heartbeater.cc:503] Master 127.0.98.62:32927 was elected leader, sending a full tablet report...
I20260926 14:47:08.935614 13631 tablet_bootstrap.cc:492] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: No bootstrap required, opened a new log
I20260926 14:47:08.935832 13631 ts_tablet_manager.cc:1397] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20260926 14:47:08.940091 13631 raft_consensus.cc:348] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accf1f4b0515453bbfb6ca19eb308255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37015 } } peers { permanent_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37619 } } peers { permanent_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43701 } }
I20260926 14:47:08.940412 13631 raft_consensus.cc:374] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:08.940474 13631 raft_consensus.cc:729] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c7bc8acbbe54bc08f9c21180ab24ef6, State: Initialized, Role: FOLLOWER
I20260926 14:47:08.940946 13631 consensus_queue.cc:260] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accf1f4b0515453bbfb6ca19eb308255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37015 } } peers { permanent_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37619 } } peers { permanent_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43701 } }
I20260926 14:47:08.941671 13464 heartbeater.cc:503] Master 127.0.98.62:32927 was elected leader, sending a full tablet report...
I20260926 14:47:08.942130 13631 ts_tablet_manager.cc:1428] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260926 14:47:09.058373 13465 tablet.cc:2369] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 14:47:09 dist-test-slave-s7k6 krb5kdc[12135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434028, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 14:47:09.072332 13330 tablet.cc:2369] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:47:09.075449 13605 tablet.cc:2369] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:47:09.180508 13636 raft_consensus.cc:482] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:47:09.180740 13636 raft_consensus.cc:504] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accf1f4b0515453bbfb6ca19eb308255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37015 } } peers { permanent_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37619 } } peers { permanent_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43701 } }
I20260926 14:47:09.181896 13636 leader_election.cc:288] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c7bc8acbbe54bc08f9c21180ab24ef6 (127.0.98.2:37619), 5964cf9e62fb491e9ac8416c9c01e6eb (127.0.98.1:43701)
I20260926 14:47:09.196888 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:09.182558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56389 (local address 127.0.98.2:37619)
0926 14:47:09.182689 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:09.182693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:09.182713 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:47:09.182934 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:09.182938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:09.182992 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:09.183039 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:09.183976 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:09.184728 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:09.194868 (+ 10140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:09.195584 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:09.195692 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:09.195774 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:09.196579 (+ 805us) server_negotiation.cc:299] Negotiation successful
0926 14:47:09.196682 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 14:47:09.197595 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:09.182127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37619 (local address 127.0.98.3:56389)
0926 14:47:09.182619 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:09.182635 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:09.182750 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:09.183192 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:09.183195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:09.183217 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:09.183817 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:09.183826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:09.185056 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:09.185060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:09.194569 (+ 9509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:09.194597 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:09.195918 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:09.195922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:09.196027 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:09.196053 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:47:09.197316 (+ 1263us) client_negotiation.cc:240] Negotiation successful
0926 14:47:09.197426 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":123,"threads_started":1}
I20260926 14:47:09.200858 13416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a83294c7b46ba9a874e4ca2626e9a" candidate_uuid: "accf1f4b0515453bbfb6ca19eb308255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" is_pre_election: true
I20260926 14:47:09.201272 13416 raft_consensus.cc:2454] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate accf1f4b0515453bbfb6ca19eb308255 in term 0.
I20260926 14:47:09.202018 13486 leader_election.cc:302] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c7bc8acbbe54bc08f9c21180ab24ef6, accf1f4b0515453bbfb6ca19eb308255; no voters:
I20260926 14:47:09.202401 13636 raft_consensus.cc:2781] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:47:09.202488 13636 raft_consensus.cc:482] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:47:09.202533 13636 raft_consensus.cc:3037] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:09.207696 13636 raft_consensus.cc:504] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accf1f4b0515453bbfb6ca19eb308255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37015 } } peers { permanent_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37619 } } peers { permanent_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43701 } }
I20260926 14:47:09.208184 13636 leader_election.cc:288] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [CANDIDATE]: Term 1 election: Requested vote from peers 6c7bc8acbbe54bc08f9c21180ab24ef6 (127.0.98.2:37619), 5964cf9e62fb491e9ac8416c9c01e6eb (127.0.98.1:43701)
I20260926 14:47:09.208959 13416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a83294c7b46ba9a874e4ca2626e9a" candidate_uuid: "accf1f4b0515453bbfb6ca19eb308255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6"
I20260926 14:47:09.209115 13416 raft_consensus.cc:3037] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:09.210615 13416 raft_consensus.cc:2454] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate accf1f4b0515453bbfb6ca19eb308255 in term 1.
I20260926 14:47:09.211113 13486 leader_election.cc:302] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c7bc8acbbe54bc08f9c21180ab24ef6, accf1f4b0515453bbfb6ca19eb308255; no voters:
I20260926 14:47:09.211372 13636 raft_consensus.cc:2781] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:47:09.211635 13636 raft_consensus.cc:686] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 1 LEADER]: Becoming Leader. State: Replica: accf1f4b0515453bbfb6ca19eb308255, State: Running, Role: LEADER
I20260926 14:47:09.212029 13636 consensus_queue.cc:237] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accf1f4b0515453bbfb6ca19eb308255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37015 } } peers { permanent_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37619 } } peers { permanent_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43701 } }
I20260926 14:47:09.215965 13130 catalog_manager.cc:5495] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 reported cstate change: term changed from 0 to 1, leader changed from <none> to accf1f4b0515453bbfb6ca19eb308255 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "accf1f4b0515453bbfb6ca19eb308255" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "accf1f4b0515453bbfb6ca19eb308255" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260926 14:47:09.222952 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:09.182241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43701 (local address 127.0.98.3:57145)
0926 14:47:09.187945 (+ 5704us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:09.187963 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:09.198869 (+ 10906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:09.199338 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:09.199341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:09.199360 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:09.199535 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:09.199542 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:09.219839 (+ 20297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:09.219843 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:09.221195 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:09.221210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:09.222444 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:09.222448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:09.222556 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:09.222583 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 14:47:09.222663 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 14:47:09.222770 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5606,"thread_start_us":76,"threads_started":1}
I20260926 14:47:09.222988 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:09.182586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57145 (local address 127.0.98.1:43701)
0926 14:47:09.182696 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:09.182701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:09.188062 (+ 5361us) server_negotiation.cc:407] Connection header received
0926 14:47:09.199048 (+ 10986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:09.199053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:09.199114 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:09.199178 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:09.199681 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:09.200421 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:09.221461 (+ 21040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:09.222127 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:09.222222 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:09.222294 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:09.222716 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 14:47:09.222860 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:47:09.224246 13267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a83294c7b46ba9a874e4ca2626e9a" candidate_uuid: "accf1f4b0515453bbfb6ca19eb308255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb"
I20260926 14:47:09.224274 13269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220a83294c7b46ba9a874e4ca2626e9a" candidate_uuid: "accf1f4b0515453bbfb6ca19eb308255" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" is_pre_election: true
I20260926 14:47:09.224587 13269 raft_consensus.cc:2454] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate accf1f4b0515453bbfb6ca19eb308255 in term 0.
I20260926 14:47:09.254392 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:09.250189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.0.98.3:37015)
0926 14:47:09.250326 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:09.250331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:09.250344 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:47:09.250424 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:09.250426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:09.250482 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:09.250575 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:47:09.250876 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:09.251426 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:09.252531 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:09.252862 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:09.252971 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:09.253459 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:47:09.253956 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:47:09.254075 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:09.254160 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 14:47:09.254223 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 14:47:09.262652 13267 raft_consensus.cc:3037] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:09.264295 13267 raft_consensus.cc:1261] T 220a83294c7b46ba9a874e4ca2626e9a P 5964cf9e62fb491e9ac8416c9c01e6eb [term 1 FOLLOWER]: Refusing update from remote peer accf1f4b0515453bbfb6ca19eb308255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:09.265122 13643 consensus_queue.cc:1035] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5964cf9e62fb491e9ac8416c9c01e6eb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:47:09.266444 13416 raft_consensus.cc:1261] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6 [term 1 FOLLOWER]: Refusing update from remote peer accf1f4b0515453bbfb6ca19eb308255: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:09.267149 13643 consensus_queue.cc:1035] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c7bc8acbbe54bc08f9c21180ab24ef6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:47:09.284062 13646 mvcc.cc:205] Tried to move back new op lower bound from 7333617783849795584 to 7333617783656611840. Current Snapshot: MvccSnapshot[applied={T|T < 7333617783849795584}]
I20260926 14:47:09.287604 13648 mvcc.cc:205] Tried to move back new op lower bound from 7333617783849795584 to 7333617783656611840. Current Snapshot: MvccSnapshot[applied={T|T < 7333617783849795584}]
I20260926 14:47:09.296957 13647 mvcc.cc:205] Tried to move back new op lower bound from 7333617783849795584 to 7333617783656611840. Current Snapshot: MvccSnapshot[applied={T|T < 7333617783849795584}]
I20260926 14:47:09.297457 13131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:47:09.297614 13131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:47:09.299986 13131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb73f098381a4d629bf750646dedbffc: Sending DeleteTablet for 3 replicas of tablet 220a83294c7b46ba9a874e4ca2626e9a
I20260926 14:47:09.300849 13398 tablet_service.cc:1514] Processing DeleteTablet for tablet 220a83294c7b46ba9a874e4ca2626e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43550
I20260926 14:47:09.300971 13538 tablet_service.cc:1514] Processing DeleteTablet for tablet 220a83294c7b46ba9a874e4ca2626e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56814
I20260926 14:47:09.301010 13263 tablet_service.cc:1514] Processing DeleteTablet for tablet 220a83294c7b46ba9a874e4ca2626e9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52894
I20260926 14:47:09.301970 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 13169
I20260926 14:47:09.306025 13655 tablet_replica.cc:330] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: stopping tablet replica
I20260926 14:47:09.306320 13655 raft_consensus.cc:2229] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:47:09.306622 13655 raft_consensus.cc:2258] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:47:09.310503 13118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43701 (error 108)
I20260926 14:47:09.313109 13655 ts_tablet_manager.cc:1905] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:09.315769 13656 tablet_replica.cc:330] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: stopping tablet replica
I20260926 14:47:09.316207 13655 ts_tablet_manager.cc:1918] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:47:09.316318 13655 log.cc:1196] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/220a83294c7b46ba9a874e4ca2626e9a
I20260926 14:47:09.316656 13655 ts_tablet_manager.cc:1939] T 220a83294c7b46ba9a874e4ca2626e9a P 6c7bc8acbbe54bc08f9c21180ab24ef6: Deleting consensus metadata
I20260926 14:47:09.317868 13656 raft_consensus.cc:2229] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:47:09.318125 13117 catalog_manager.cc:4841] TS 6c7bc8acbbe54bc08f9c21180ab24ef6 (127.0.98.2:37619): tablet 220a83294c7b46ba9a874e4ca2626e9a (table test-table [id=adfe46ff763e493cbb53cff224d25421]) successfully deleted
I20260926 14:47:09.318648 13656 raft_consensus.cc:2258] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 14:47:09.320902 13625 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:47:09.311405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43701 (local address 127.0.0.1:52902)
0926 14:47:09.311575 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:09.311592 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:09.311700 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:09.320758 (+ 9058us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43701: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":69}
W20260926 14:47:09.321165 13118 catalog_manager.cc:4584] TS 5964cf9e62fb491e9ac8416c9c01e6eb (127.0.98.1:43701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 220a83294c7b46ba9a874e4ca2626e9a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43701: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:47:09.322237 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 13331
I20260926 14:47:09.327802 13656 ts_tablet_manager.cc:1905] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:09.330107 13656 ts_tablet_manager.cc:1918] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:47:09.330210 13656 log.cc:1196] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/220a83294c7b46ba9a874e4ca2626e9a
I20260926 14:47:09.330463 13656 ts_tablet_manager.cc:1939] T 220a83294c7b46ba9a874e4ca2626e9a P accf1f4b0515453bbfb6ca19eb308255: Deleting consensus metadata
I20260926 14:47:09.331718 13118 catalog_manager.cc:4841] TS accf1f4b0515453bbfb6ca19eb308255 (127.0.98.3:37015): tablet 220a83294c7b46ba9a874e4ca2626e9a (table test-table [id=adfe46ff763e493cbb53cff224d25421]) successfully deleted
I20260926 14:47:09.336344 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 13467
I20260926 14:47:09.343776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 13098
I20260926 14:47:09.351841 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 14:47:09.686951 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 14:47:09.687062 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 14:47:10.021927 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 14:47:10.022 UTC [12165] LOG: received smart shutdown request
2026-09-26 14:47:10.025 UTC [12165] LOG: background worker "logical replication launcher" (PID 12174) exited with exit code 1
2026-09-26 14:47:10.025 UTC [12169] LOG: shutting down
2026-09-26 14:47:10.189 UTC [12165] LOG: database system is shut down
2026-09-26T14:47:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48755 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:47:10 dist-test-slave-s7k6 krb5kdc[13669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:47:10 dist-test-slave-s7k6 krb5kdc[13669](info): set up 2 sockets
Sep 26 14:47:10 dist-test-slave-s7k6 krb5kdc[13669](info): commencing operation
krb5kdc: starting...
W20260926 14:47:11.242614 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:47:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:47:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:47:11.357897 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40493
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40493,127.0.98.61:42619,127.0.98.60:43707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:11.469349 13682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:11.469673 13682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:11.469732 13682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:11.473969 13682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:47:11.474053 13682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:11.474081 13682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:11.474112 13682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:47:11.474134 13682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:47:11.479897 13682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40493,127.0.98.61:42619,127.0.98.60:43707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:11.481053 13682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:11.482155 13682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:11.490553 13690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:11.490553 13688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:11.490553 13687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:11.490993 13682 server_base.cc:1034] running on GCE node
I20260926 14:47:11.491568 13682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:11.492746 13682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:11.493911 13682 hybrid_clock.cc:648] HybridClock initialized: now 1790434031493918 us; error 62 us; skew 500 ppm
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:11.496694 13682 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:47:11.497573 13682 webserver.cc:460] Webserver started at http://127.0.98.62:41931/ using document root <none> and password file <none>
I20260926 14:47:11.498093 13682 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:11.498152 13682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:11.498311 13682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:11.500349 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "111247579b914f4cbfc9fa94b7ca44bc"
format_stamp: "Formatted at 2026-09-26 14:47:11 on dist-test-slave-s7k6"
server_key: "2e0225037742fbda8774b976c32e60e1"
server_key_iv: "7fd5dab410dc3bb7847154a3d3d6930c"
server_key_version: "encryptionkey@0"
I20260926 14:47:11.500859 13682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "111247579b914f4cbfc9fa94b7ca44bc"
format_stamp: "Formatted at 2026-09-26 14:47:11 on dist-test-slave-s7k6"
server_key: "2e0225037742fbda8774b976c32e60e1"
server_key_iv: "7fd5dab410dc3bb7847154a3d3d6930c"
server_key_version: "encryptionkey@0"
I20260926 14:47:11.504621 13682 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:47:11.507122 13697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:11.508450 13682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:47:11.508566 13682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "111247579b914f4cbfc9fa94b7ca44bc"
format_stamp: "Formatted at 2026-09-26 14:47:11 on dist-test-slave-s7k6"
server_key: "2e0225037742fbda8774b976c32e60e1"
server_key_iv: "7fd5dab410dc3bb7847154a3d3d6930c"
server_key_version: "encryptionkey@0"
I20260926 14:47:11.508701 13682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:11.525867 13682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:11.529129 13682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:11.529311 13682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:11.536293 13682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40493
I20260926 14:47:11.536334 13749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40493 every 8 connection(s)
I20260926 14:47:11.537277 13682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:47:11.540146 13750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:11.543887 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:47:11.544471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 13682
I20260926 14:47:11.544597 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:47:11.544843 392 external_mini_cluster.cc:1428] Setting key 04280f295d68d1f0ad5e935ce9044acb
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:11.560817 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.546226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40493 (local address 127.0.0.1:59272)
0926 14:47:11.546839 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.546878 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:11.547721 (+ 843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:11.548421 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:11.548426 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:11.548866 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:11.549163 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.549184 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.550977 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.550980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.551765 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.551776 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.552676 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.552679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.552781 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.553409 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:11.553430 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:11.555709 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:11.557798 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:11.557806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:11.557818 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:11.558037 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:11.558340 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:11.558343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:11.558345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:11.558453 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:11.559082 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:11.559089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:11.559239 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 14:47:11.559394 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 14:47:11.559771 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":101,"spinlock_wait_cycles":38656,"thread_start_us":162,"threads_started":1}
I20260926 14:47:11.560817 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.546225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.0.98.62:40493)
0926 14:47:11.546664 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:11.546684 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:11.546982 (+ 298us) server_negotiation.cc:407] Connection header received
0926 14:47:11.548026 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:11.548030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:11.548230 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:11.548398 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:11.549361 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.550832 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.551902 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.552505 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.552693 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.555862 (+ 3169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:11.555868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:11.555879 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:11.555900 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:11.557698 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.558141 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.558147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.558149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.558256 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.558533 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.558536 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.558538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.558981 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:11.559112 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:11.559466 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 14:47:11.559713 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":288,"spinlock_wait_cycles":2816,"thread_start_us":111,"threads_started":1}
I20260926 14:47:11.563463 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:47:11.564260 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.563962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42619 (local address 127.0.0.1:40618)
0926 14:47:11.564074 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.564196 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:47:11.564478 13701 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42619: connect: Connection refused (error 111)
I20260926 14:47:11.565337 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.565104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42619 (local address 127.0.0.1:40634)
0926 14:47:11.565221 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.565273 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 14:47:11.565457 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42619: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42619: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 14:47:11.569741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42619
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40493,127.0.98.61:42619,127.0.98.60:43707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:11.614682 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } attempt: 1
I20260926 14:47:11.615592 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.615256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42619 (local address 127.0.0.1:40650)
0926 14:47:11.615424 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.615497 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 14:47:11.616278 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.616066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42619 (local address 127.0.0.1:40654)
0926 14:47:11.616160 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.616212 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 14:47:11.616405 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42619: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42619: connect: Connection refused (error 111)
I20260926 14:47:11.684587 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } attempt: 2
I20260926 14:47:11.685461 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.685134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42619 (local address 127.0.0.1:40668)
0926 14:47:11.685295 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.685370 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 14:47:11.685403 13757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:11.685770 13757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:11.685845 13757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 14:47:11.686192 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.685972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42619 (local address 127.0.0.1:40684)
0926 14:47:11.686076 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.686126 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 14:47:11.686307 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42619: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42619: connect: Connection refused (error 111)
W20260926 14:47:11.690191 13757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:47:11.690263 13757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:11.690281 13757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:11.690297 13757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:47:11.690317 13757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:47:11.695851 13757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40493,127.0.98.61:42619,127.0.98.60:43707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:11.696911 13757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:11.697897 13757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:11.706012 13762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:11.706044 13763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:11.706091 13757 server_base.cc:1034] running on GCE node
W20260926 14:47:11.706012 13765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:11.706854 13757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:11.707952 13757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:11.709110 13757 hybrid_clock.cc:648] HybridClock initialized: now 1790434031709091 us; error 40 us; skew 500 ppm
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:11.711898 13757 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 14:47:11.712908 13757 webserver.cc:460] Webserver started at http://127.0.98.61:32963/ using document root <none> and password file <none>
I20260926 14:47:11.713456 13757 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:11.713527 13757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:11.713742 13757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:11.715883 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/data/instance:
uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980"
format_stamp: "Formatted at 2026-09-26 14:47:11 on dist-test-slave-s7k6"
server_key: "a327dd69d9cc5c913c6f121fe35f9da3"
server_key_iv: "4af11eecd95f008d3869fe9b89f064b7"
server_key_version: "encryptionkey@0"
I20260926 14:47:11.716423 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/wal/instance:
uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980"
format_stamp: "Formatted at 2026-09-26 14:47:11 on dist-test-slave-s7k6"
server_key: "a327dd69d9cc5c913c6f121fe35f9da3"
server_key_iv: "4af11eecd95f008d3869fe9b89f064b7"
server_key_version: "encryptionkey@0"
I20260926 14:47:11.720204 13757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:47:11.722658 13772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:11.723973 13757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:47:11.724102 13757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/wal
uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980"
format_stamp: "Formatted at 2026-09-26 14:47:11 on dist-test-slave-s7k6"
server_key: "a327dd69d9cc5c913c6f121fe35f9da3"
server_key_iv: "4af11eecd95f008d3869fe9b89f064b7"
server_key_version: "encryptionkey@0"
I20260926 14:47:11.724206 13757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:11.749852 13757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:11.753079 13757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:11.753259 13757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:11.760251 13757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42619
I20260926 14:47:11.760269 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42619 every 8 connection(s)
I20260926 14:47:11.761204 13757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
I20260926 14:47:11.764108 13825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:11.765947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 13757
I20260926 14:47:11.766081 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-1/wal/instance
I20260926 14:47:11.766362 392 external_mini_cluster.cc:1428] Setting key 890df743f3e676bb16453835c975b789
I20260926 14:47:11.768648 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 14:47:11.777457 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } attempt: 3
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:11.783692 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.770671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.0.98.62:40493)
0926 14:47:11.770863 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:11.770868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:11.771781 (+ 913us) server_negotiation.cc:407] Connection header received
0926 14:47:11.772734 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:11.772738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:11.772796 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:11.772899 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:11.774072 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.774596 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.775876 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.776187 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.776294 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.778832 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:11.778836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:11.778839 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:11.778864 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:11.780927 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.781545 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.781548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.781550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.781638 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.782000 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.782003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.782005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.782226 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:11.782329 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:11.782919 (+ 590us) server_negotiation.cc:299] Negotiation successful
0926 14:47:11.783047 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 14:47:11.784252 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.771124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40493 (local address 127.0.0.1:59276)
0926 14:47:11.771603 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.771657 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:11.772506 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:11.773046 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:11.773055 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:11.773385 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:11.773930 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.773946 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.774734 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.774738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.775743 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.775753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.776353 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.776356 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.776501 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.777059 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:11.777075 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:11.778658 (+ 1583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:11.781033 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:11.781038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:11.781053 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:11.781438 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:11.781733 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:11.781736 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:11.781738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:11.781904 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:11.782331 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:11.782337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:11.782625 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 14:47:11.782855 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 14:47:11.783180 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":112,"threads_started":1}
I20260926 14:47:11.785957 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:47:11.788035 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.777919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42619 (local address 127.0.0.1:40700)
0926 14:47:11.778105 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.778119 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:11.778193 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:11.779292 (+ 1099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:11.779295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:11.779452 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:11.779583 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.779590 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.780725 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.780728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.781429 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.781438 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.782152 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.782156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.782257 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.782717 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:11.782747 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:11.784221 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:11.786332 (+ 2111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:11.786335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:11.786337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:11.786679 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:11.786954 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:11.786957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:11.786959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:11.787047 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:11.787515 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:11.787517 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:11.787645 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 14:47:11.787744 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 14:47:11.787876 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 14:47:11.788221 13830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.778037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40700 (local address 127.0.98.61:42619)
0926 14:47:11.778929 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:11.778938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:11.778957 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:47:11.778987 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:11.778991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:11.779165 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:11.779296 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:11.779701 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.780562 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.781570 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.782048 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.782129 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.784399 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:11.784412 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:11.784426 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:11.784454 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:11.786226 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.786782 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.786788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.786790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.786869 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.787135 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.787138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.787139 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.787416 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:11.787544 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:11.787853 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 14:47:11.787987 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":812,"thread_start_us":134,"threads_started":1}
I20260926 14:47:11.790383 13750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 14:47:11.791222 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.790894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40634)
0926 14:47:11.791078 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.791141 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 14:47:11.791945 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.791685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40650)
0926 14:47:11.791817 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.791874 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260926 14:47:11.792085 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43707: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 14:47:11.793774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43707
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40493,127.0.98.61:42619,127.0.98.60:43707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:11.794615 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.786441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42619 (local address 127.0.0.1:40704)
0926 14:47:11.786537 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.786552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:11.786624 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:11.787040 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:11.787043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:11.787212 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:11.787351 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.787357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.788221 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.788224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.788837 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.788846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.789236 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.789240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.789392 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.789867 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:11.789881 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:11.791355 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:11.793201 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:11.793204 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:11.793206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:11.793383 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:11.793634 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:11.793637 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:11.793639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:11.793712 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:11.794093 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:11.794096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:11.794185 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 14:47:11.794267 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 14:47:11.794410 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":23}
I20260926 14:47:11.794646 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.786545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40704 (local address 127.0.98.61:42619)
0926 14:47:11.786858 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:11.786865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:11.786876 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:47:11.786904 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:11.786906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:11.786940 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:11.787008 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:11.787455 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.788109 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.788936 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.789145 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.789213 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.791532 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:11.791536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:11.791539 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:11.791564 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:11.793101 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.793475 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.793479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.793481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.793558 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.793785 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.793788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.793790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.794012 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:11.794094 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:11.794319 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 14:47:11.794468 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
I20260926 14:47:11.795454 13825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:47:11.796113 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.795872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40652)
0926 14:47:11.795976 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.796054 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:47:11.796373 13774 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
I20260926 14:47:11.797240 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.797044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40654)
0926 14:47:11.797147 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.797188 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:47:11.797389 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43707: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
I20260926 14:47:11.823236 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } attempt: 1
I20260926 14:47:11.824052 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.823753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40660)
0926 14:47:11.823904 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.823967 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 14:47:11.824774 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.824584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40674)
0926 14:47:11.824669 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.824714 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 14:47:11.824892 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43707: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
I20260926 14:47:11.846691 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } attempt: 1
I20260926 14:47:11.847606 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.847307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40688)
0926 14:47:11.847463 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.847525 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 14:47:11.848311 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.848097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40692)
0926 14:47:11.848199 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.848251 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:47:11.848515 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43707: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
I20260926 14:47:11.900055 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } attempt: 2
I20260926 14:47:11.900869 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.900575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40696)
0926 14:47:11.900727 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.900790 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 14:47:11.901594 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.901407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40706)
0926 14:47:11.901491 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.901537 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 14:47:11.901738 13750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43707: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
W20260926 14:47:11.907115 13832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:11.907418 13832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:11.907467 13832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:11.911541 13832 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:47:11.911604 13832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:11.911623 13832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:11.911639 13832 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:47:11.911671 13832 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:47:11.916697 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } attempt: 2
I20260926 14:47:11.917478 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.917203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40716)
0926 14:47:11.917340 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.917399 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 14:47:11.917493 13832 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40493,127.0.98.61:42619,127.0.98.60:43707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:11.918154 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.917932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40718)
0926 14:47:11.918043 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.918087 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 14:47:11.918308 13825 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:43707: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43707: connect: Connection refused (error 111)
I20260926 14:47:11.918836 13832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:11.919862 13832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:11.928061 13838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:11.928046 13841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:11.928049 13839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:11.928514 13832 server_base.cc:1034] running on GCE node
I20260926 14:47:11.929087 13832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:11.930212 13832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:11.931392 13832 hybrid_clock.cc:648] HybridClock initialized: now 1790434031931366 us; error 35 us; skew 500 ppm
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:11.934190 13832 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 14:47:11.935079 13832 webserver.cc:460] Webserver started at http://127.0.98.60:41977/ using document root <none> and password file <none>
I20260926 14:47:11.935590 13832 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:11.935654 13832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:11.935829 13832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:11.937835 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/data/instance:
uuid: "c01ef8b42c15411f99e276643586d176"
format_stamp: "Formatted at 2026-09-26 14:47:11 on dist-test-slave-s7k6"
server_key: "6520a27ba8ebf8b175fcee222603b6d3"
server_key_iv: "04dd0383b086d3aea7f1a212cccadc35"
server_key_version: "encryptionkey@0"
I20260926 14:47:11.938335 13832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/wal/instance:
uuid: "c01ef8b42c15411f99e276643586d176"
format_stamp: "Formatted at 2026-09-26 14:47:11 on dist-test-slave-s7k6"
server_key: "6520a27ba8ebf8b175fcee222603b6d3"
server_key_iv: "04dd0383b086d3aea7f1a212cccadc35"
server_key_version: "encryptionkey@0"
I20260926 14:47:11.941957 13832 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 14:47:11.944491 13848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:11.945741 13832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:47:11.945842 13832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/wal
uuid: "c01ef8b42c15411f99e276643586d176"
format_stamp: "Formatted at 2026-09-26 14:47:11 on dist-test-slave-s7k6"
server_key: "6520a27ba8ebf8b175fcee222603b6d3"
server_key_iv: "04dd0383b086d3aea7f1a212cccadc35"
server_key_version: "encryptionkey@0"
I20260926 14:47:11.945930 13832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:11.960268 13832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:11.963464 13832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:11.963622 13832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:11.970634 13832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43707
I20260926 14:47:11.970659 13900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43707 every 8 connection(s)
I20260926 14:47:11.971634 13832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
I20260926 14:47:11.974395 13901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:11.978217 13901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:47:11.980595 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 13832
I20260926 14:47:11.980718 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790433872429009-392-0/minicluster-data/master-2/wal/instance
I20260926 14:47:11.981014 392 external_mini_cluster.cc:1428] Setting key 4f0a885182c1d29b5fd6c4080c299cf9
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 14:47:11.991681 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.980118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.0.98.62:40493)
0926 14:47:11.980270 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:11.980275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:11.981308 (+ 1033us) server_negotiation.cc:407] Connection header received
0926 14:47:11.982464 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:11.982468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:11.982525 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:11.982595 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:11.983524 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.984024 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.984977 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.985191 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.985279 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.987873 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:11.987878 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:11.987880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:11.987908 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:11.989360 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.989975 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.989977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.989979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.990050 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.990403 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.990406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.990408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.990647 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:11.990738 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:11.991350 (+ 612us) server_negotiation.cc:299] Negotiation successful
0926 14:47:11.991480 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:47:11.992534 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.980577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40493 (local address 127.0.0.1:59282)
0926 14:47:11.981134 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.981203 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:11.982263 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:11.982674 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:11.982681 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:11.983072 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:11.983384 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.983399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.984120 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.984124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.984865 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.984874 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.985326 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.985329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.985548 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.986119 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:11.986140 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:11.987719 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:11.989469 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:11.989476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:11.989487 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:11.989884 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:11.990130 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:11.990134 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:11.990136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:11.990309 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:11.990824 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:11.990831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:11.991081 (+ 250us) client_negotiation.cc:769] Sending connection context
0926 14:47:11.991255 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 14:47:11.991585 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":27,"thread_start_us":160,"threads_started":1}
I20260926 14:47:11.992547 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.981644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.0.98.60:43707)
0926 14:47:11.982005 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:11.982015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:11.982049 (+ 34us) server_negotiation.cc:407] Connection header received
0926 14:47:11.982304 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:11.982320 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:11.982511 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:11.982695 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:11.983550 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.984419 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.985267 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.985650 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.985751 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.988314 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:11.988320 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:11.988330 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:11.988351 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:11.989904 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.990346 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.990351 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.990352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.990443 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:11.990803 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:11.990806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:11.990808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:11.991099 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:11.991259 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:11.991504 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 14:47:11.991644 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":285,"thread_start_us":122,"threads_started":1}
I20260926 14:47:11.993850 13901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 14:47:11 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:12.000909 13750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } attempt: 3
I20260926 14:47:12.002933 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.994302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42619 (local address 127.0.0.1:40714)
0926 14:47:11.994661 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:11.994674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:11.994773 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:11.995103 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:11.995106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:11.995280 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:11.995397 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.995403 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.996339 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.996342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.996954 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:11.996963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.997517 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.997521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:11.997621 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.998154 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:11.998166 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:11.999441 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:12.001311 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:12.001314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:12.001316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:12.001550 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:12.001894 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:12.001897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:12.001898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:12.001985 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:12.002419 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:12.002421 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:12.002502 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 14:47:12.002604 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:47:12.002738 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295}
I20260926 14:47:12.003010 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.994429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.0.98.61:42619)
0926 14:47:11.994591 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:11.994596 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:11.994730 (+ 134us) server_negotiation.cc:407] Connection header received
0926 14:47:11.994851 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:11.994854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:11.994903 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:11.994973 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:11.995517 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.996220 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.997070 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.997416 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.997496 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:11.999567 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:11.999570 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:11.999573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:11.999598 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:12.001207 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.001697 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.001701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.001703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.001799 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.002078 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.002081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.002083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.002296 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:12.002381 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:12.002693 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 14:47:12.002844 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 14:47:12.003834 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:11.994986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.0.98.62:40493)
0926 14:47:11.995098 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:11.995102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:11.995113 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:47:11.995174 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:11.995176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:11.995216 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:11.995292 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:11.995955 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.996533 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.997398 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:11.997722 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:11.997825 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:12.000078 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:12.000082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:12.000085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:12.000111 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:12.001702 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.002132 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.002136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.002137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.002207 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.002452 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.002455 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.002457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.002649 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:12.003136 (+ 487us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:12.003440 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 14:47:12.003696 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:47:12.004346 13901 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:47:12 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 14:47:12.009460 13825 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } attempt: 3
Sep 26 14:47:12 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 14:47:12 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 14:47:12.011653 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:12.001398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40744)
0926 14:47:12.001501 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:12.001515 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:12.001584 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:12.001891 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:12.001894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:12.002035 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:12.002135 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:12.002142 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.003204 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.003208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:12.003902 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:12.003913 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.004652 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.004655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:12.004730 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:12.005193 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:12.005206 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:12.006583 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:12.010029 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:12.010032 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:12.010034 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:12.010289 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:12.010588 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:12.010591 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:12.010593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:12.010680 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:12.011180 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:12.011183 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:12.011278 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 14:47:12.011359 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 14:47:12.011488 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 14:47:12.012154 13905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:12.001570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.0.98.60:43707)
0926 14:47:12.001683 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:12.001687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:12.001698 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:47:12.001728 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:12.001731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:12.001769 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:12.002252 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:12.002336 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.003071 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.004196 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.004531 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.005093 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:12.006721 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:12.006725 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:12.006728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:12.006749 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:12.009901 (+ 3152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.010417 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.010420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.010422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.010494 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.010835 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.010837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.010839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.011049 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:12.011826 (+ 777us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:12.011898 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 14:47:12.012003 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:47:12.014840 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:12.004885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40754 (local address 127.0.98.60:43707)
0926 14:47:12.005311 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:12.005320 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:12.005329 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:47:12.005354 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:12.005357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:12.005389 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:12.005473 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:12.005850 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.007110 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.008143 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.008422 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.008512 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:12.010839 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:12.010842 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:12.010845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:12.010867 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:12.012926 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.013332 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.013335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.013337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.013413 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.013625 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.013627 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.013629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.013840 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:12.013922 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:12.014530 (+ 608us) server_negotiation.cc:299] Negotiation successful
0926 14:47:12.014629 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":120,"threads_started":1}
I20260926 14:47:12.015120 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:12.004767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40754)
0926 14:47:12.004860 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:12.004874 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:12.004941 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:12.005496 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:12.005499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:12.005656 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:12.005745 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:12.005750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.007236 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.007240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:12.008029 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:12.008037 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.008522 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.008526 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:12.008600 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:12.008882 (+ 282us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:12.008896 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:12.010687 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:12.013028 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:12.013031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:12.013032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:12.013245 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:12.013488 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:12.013491 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:12.013493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:12.013556 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:12.013953 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:12.013955 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:12.014036 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 14:47:12.014122 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 14:47:12.014240 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 14:47:12.015163 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:12.004980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.0.98.61:42619)
0926 14:47:12.005076 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:12.005080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:12.005233 (+ 153us) server_negotiation.cc:407] Connection header received
0926 14:47:12.005340 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:12.005343 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:12.005387 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:12.005464 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:12.006439 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.007151 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.008191 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.008505 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.008574 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:12.011201 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:12.011204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:12.011206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:12.011228 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:12.013307 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.013778 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.013782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.013783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.013861 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.014154 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.014157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.014159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.014354 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:12.014516 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:12.014890 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 14:47:12.015002 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 14:47:12 dist-test-slave-s7k6 krb5kdc[13669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434031, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 14:47:12.016556 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc: Bootstrap starting.
I20260926 14:47:12.017590 13901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176: Bootstrap starting.
I20260926 14:47:12.019111 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:12.010078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43707 (local address 127.0.0.1:40770)
0926 14:47:12.010273 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:12.010288 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:12.010358 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:12.010666 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:12.010670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:12.010860 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:12.010956 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:12.010963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.012207 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.012211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:12.013029 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:12.013039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.014017 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.014021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:12.014104 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:12.014495 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:12.014508 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:12.015827 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:12.017637 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:12.017640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:12.017642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:12.017867 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:12.018139 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:12.018142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:12.018144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:12.018239 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:12.018621 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:12.018624 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:12.018718 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 14:47:12.018844 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 14:47:12.018968 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260926 14:47:12.019651 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:12.010132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40770 (local address 127.0.98.60:43707)
0926 14:47:12.010453 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:12.010461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:12.010472 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:47:12.010501 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:12.010504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:12.010539 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:12.011063 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:12.011194 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.012062 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.013358 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.013689 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.014301 (+ 612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:12.015960 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:12.015963 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:12.015966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:12.015989 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:12.017521 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.017965 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.017968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.017970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.018048 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.018321 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.018324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.018326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.018514 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:12.018610 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:12.019399 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 14:47:12.019504 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":110,"threads_started":1}
I20260926 14:47:12.020123 13901 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:12.020509 13750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:12.020948 13901 log.cc:824] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:12.022398 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:47:12.022428 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980: Bootstrap starting.
I20260926 14:47:12.022961 13901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176: No bootstrap required, opened a new log
I20260926 14:47:12.024488 13750 log.cc:824] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:12.024945 13825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:12.025719 13825 log.cc:824] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:12.027242 13901 raft_consensus.cc:348] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } }
I20260926 14:47:12.027503 13901 raft_consensus.cc:374] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:12.027557 13901 raft_consensus.cc:729] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c01ef8b42c15411f99e276643586d176, State: Initialized, Role: FOLLOWER
I20260926 14:47:12.027671 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980: No bootstrap required, opened a new log
I20260926 14:47:12.028156 13901 consensus_queue.cc:260] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } }
I20260926 14:47:12.028710 13750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc: No bootstrap required, opened a new log
I20260926 14:47:12.029546 13901 sys_catalog.cc:564] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:47:12.031814 13825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } }
I20260926 14:47:12.032052 13825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:12.032101 13825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ebaf6d99b6d46c6a73c29f73a2c0980, State: Initialized, Role: FOLLOWER
I20260926 14:47:12.032626 13825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } }
I20260926 14:47:12.033993 13825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 14:47:12.035856 13928 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:47:12.035930 13928 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:47:12.036532 13750 raft_consensus.cc:348] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } }
I20260926 14:47:12.034139 13832 master_runner.cc:186] Error getting master registration for 127.0.98.62:40493: OK, instance_id { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" instance_seqno: 1790434031534302 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:47:12.037606 13832 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:47:12.038640 13750 raft_consensus.cc:374] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:12.038708 13750 raft_consensus.cc:729] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111247579b914f4cbfc9fa94b7ca44bc, State: Initialized, Role: FOLLOWER
I20260926 14:47:12.039285 13750 consensus_queue.cc:260] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } }
I20260926 14:47:12.039909 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } } }
I20260926 14:47:12.040027 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:47:12.041115 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:12.024541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59292 (local address 127.0.98.62:40493)
0926 14:47:12.025091 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:12.025095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:12.025321 (+ 226us) server_negotiation.cc:407] Connection header received
0926 14:47:12.025489 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:12.025492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:12.025537 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:12.025582 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:12.026562 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.027313 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.034019 (+ 6706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.034335 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.034434 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:12.036102 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:12.036106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:12.036109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:12.036134 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:12.037794 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.038256 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.038259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.038261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.038345 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.039044 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.039047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.039050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.040356 (+ 1306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:12.040496 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:12.040836 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 14:47:12.040960 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403}
I20260926 14:47:12.042382 13750 sys_catalog.cc:564] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:47:12.042739 13831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:12.025078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40742 (local address 127.0.98.61:42619)
0926 14:47:12.025183 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:12.025187 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:12.027127 (+ 1940us) server_negotiation.cc:407] Connection header received
0926 14:47:12.027335 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:12.027339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:12.027400 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:12.027448 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:12.028495 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.029211 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.030375 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.030689 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.030802 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:12.038034 (+ 7232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:12.038039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:12.038043 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:12.038073 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:12.039962 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.041141 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.041145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.041147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.041250 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.041770 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.041774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.041776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.041986 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:12.042082 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:12.042476 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 14:47:12.042590 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:47:12.039636 13757 master_runner.cc:186] Error getting master registration for 127.0.98.62:40493: OK, instance_id { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" instance_seqno: 1790434031534302 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:47:12.043015 13757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:47:12.043959 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } } }
I20260926 14:47:12.044106 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 14:47:12.047807 13942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:47:12.047883 13942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:47:12.048514 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:12.032085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.0.98.60:43707)
0926 14:47:12.032247 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:12.032252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:12.035009 (+ 2757us) server_negotiation.cc:407] Connection header received
0926 14:47:12.035195 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:12.035199 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:12.035246 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:12.035295 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:12.037900 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.038710 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.042858 (+ 4148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:12.043236 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:12.043325 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:12.045003 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:12.045007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:12.045009 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:12.045037 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:12.046605 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.047115 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.047119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.047121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.047208 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:12.047510 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:12.047514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:12.047516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:12.047754 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:12.047885 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:12.048239 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 14:47:12.048367 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260926 14:47:12.050966 13954 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:47:12.051043 13954 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:47:12.051484 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } } }
I20260926 14:47:12.051601 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:47:12.052629 13682 master_runner.cc:263] No leader master found from master 111247579b914f4cbfc9fa94b7ca44bc
I20260926 14:47:12.052695 13682 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:47:12.167891 13917 raft_consensus.cc:482] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:47:12.168146 13917 raft_consensus.cc:504] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } }
I20260926 14:47:12.169446 13917 leader_election.cc:288] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 111247579b914f4cbfc9fa94b7ca44bc (127.0.98.62:40493), 6ebaf6d99b6d46c6a73c29f73a2c0980 (127.0.98.61:42619)
I20260926 14:47:12.170585 13799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c01ef8b42c15411f99e276643586d176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" is_pre_election: true
I20260926 14:47:12.171052 13799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c01ef8b42c15411f99e276643586d176 in term 0.
I20260926 14:47:12.171715 13852 leader_election.cc:302] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ebaf6d99b6d46c6a73c29f73a2c0980, c01ef8b42c15411f99e276643586d176; no voters:
I20260926 14:47:12.172142 13917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:47:12.172232 13917 raft_consensus.cc:482] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:47:12.172277 13917 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:12.173607 13917 raft_consensus.cc:504] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } }
I20260926 14:47:12.173959 13917 leader_election.cc:288] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [CANDIDATE]: Term 1 election: Requested vote from peers 111247579b914f4cbfc9fa94b7ca44bc (127.0.98.62:40493), 6ebaf6d99b6d46c6a73c29f73a2c0980 (127.0.98.61:42619)
I20260926 14:47:12.174669 13799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c01ef8b42c15411f99e276643586d176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980"
I20260926 14:47:12.174849 13799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:12.176369 13799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c01ef8b42c15411f99e276643586d176 in term 1.
I20260926 14:47:12.176795 13852 leader_election.cc:302] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ebaf6d99b6d46c6a73c29f73a2c0980, c01ef8b42c15411f99e276643586d176; no voters:
I20260926 14:47:12.177616 13724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c01ef8b42c15411f99e276643586d176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111247579b914f4cbfc9fa94b7ca44bc" is_pre_election: true
I20260926 14:47:12.177937 13724 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c01ef8b42c15411f99e276643586d176 in term 0.
I20260926 14:47:12.177618 13723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c01ef8b42c15411f99e276643586d176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111247579b914f4cbfc9fa94b7ca44bc"
I20260926 14:47:12.178200 13723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:12.179551 13723 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c01ef8b42c15411f99e276643586d176 in term 1.
I20260926 14:47:12.180280 13917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:47:12.180544 13917 raft_consensus.cc:686] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [term 1 LEADER]: Becoming Leader. State: Replica: c01ef8b42c15411f99e276643586d176, State: Running, Role: LEADER
I20260926 14:47:12.180905 13917 consensus_queue.cc:237] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } }
I20260926 14:47:12.182724 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c01ef8b42c15411f99e276643586d176. Latest consensus state: current_term: 1 leader_uuid: "c01ef8b42c15411f99e276643586d176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } } }
I20260926 14:47:12.185704 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:12.186185 13959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:47:12.188094 13959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:47:12.195837 13799 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [term 1 FOLLOWER]: Refusing update from remote peer c01ef8b42c15411f99e276643586d176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:12.196761 13957 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:47:12.202035 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c01ef8b42c15411f99e276643586d176. Latest consensus state: current_term: 1 leader_uuid: "c01ef8b42c15411f99e276643586d176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } } }
I20260926 14:47:12.202198 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:47:12.206072 13723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [term 1 FOLLOWER]: Refusing update from remote peer c01ef8b42c15411f99e276643586d176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:12.206863 13957 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:47:12.210749 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c01ef8b42c15411f99e276643586d176. Latest consensus state: current_term: 1 leader_uuid: "c01ef8b42c15411f99e276643586d176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } } }
I20260926 14:47:12.210899 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:47:12.213887 13917 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c01ef8b42c15411f99e276643586d176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } } }
I20260926 14:47:12.214021 13917 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:12.214629 13943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c01ef8b42c15411f99e276643586d176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } } }
I20260926 14:47:12.214735 13943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:47:12.215338 13960 mvcc.cc:205] Tried to move back new op lower bound from 7333617795863465984 to 7333617795816304640. Current Snapshot: MvccSnapshot[applied={T|T < 7333617795863465984}]
I20260926 14:47:12.217344 13957 sys_catalog.cc:455] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c01ef8b42c15411f99e276643586d176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } } }
I20260926 14:47:12.217478 13957 sys_catalog.cc:458] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176 [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:12.218402 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c01ef8b42c15411f99e276643586d176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "111247579b914f4cbfc9fa94b7ca44bc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40493 } } peers { permanent_uuid: "6ebaf6d99b6d46c6a73c29f73a2c0980" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42619 } } peers { permanent_uuid: "c01ef8b42c15411f99e276643586d176" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43707 } } }
I20260926 14:47:12.218524 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:47:12.220244 13959 catalog_manager.cc:1347] Generated new cluster ID: 44610f3b53cb45dcaf69bfdbc51b0718
I20260926 14:47:12.220302 13959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:47:12.228644 13961 mvcc.cc:205] Tried to move back new op lower bound from 7333617795863465984 to 7333617795816304640. Current Snapshot: MvccSnapshot[applied={T|T < 7333617795863465984}]
I20260926 14:47:12.247715 13959 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:47:12.248963 13959 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:47:12.255553 13959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c01ef8b42c15411f99e276643586d176: Generated new TSK 0
I20260926 14:47:12.256407 13959 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:47:13.049306 13942 catalog_manager.cc:1259] Loaded cluster ID: 44610f3b53cb45dcaf69bfdbc51b0718
I20260926 14:47:13.049396 13942 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980: loading cluster ID for follower catalog manager: success
I20260926 14:47:13.051110 13942 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980: acquiring CA information for follower catalog manager: success
I20260926 14:47:13.052168 13954 catalog_manager.cc:1259] Loaded cluster ID: 44610f3b53cb45dcaf69bfdbc51b0718
I20260926 14:47:13.052228 13954 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc: loading cluster ID for follower catalog manager: success
I20260926 14:47:13.052328 13942 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ebaf6d99b6d46c6a73c29f73a2c0980: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:47:13.053722 13954 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc: acquiring CA information for follower catalog manager: success
I20260926 14:47:13.054898 13954 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 111247579b914f4cbfc9fa94b7ca44bc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:47:13.316172 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 13682
I20260926 14:47:13.321750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 13757
I20260926 14:47:13.326722 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 13832
2026-09-26T14:47:13Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3039 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:47:13 dist-test-slave-s7k6 krb5kdc[13973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:47:13 dist-test-slave-s7k6 krb5kdc[13973](info): set up 2 sockets
Sep 26 14:47:13 dist-test-slave-s7k6 krb5kdc[13973](info): commencing operation
krb5kdc: starting...
W20260926 14:47:14.294883 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:47:14 dist-test-slave-s7k6 krb5kdc[13973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:47:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:47:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:47:14.413411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42543
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33545
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:14.523751 13986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:14.524029 13986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:14.524070 13986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:14.528211 13986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:47:14.528275 13986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:14.528294 13986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:14.528311 13986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:47:14.528323 13986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:47:14.533747 13986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33545
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:14.534750 13986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:14.535749 13986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:14.544059 13992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:14.544077 13991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:14.544148 13994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:14.544376 13986 server_base.cc:1034] running on GCE node
I20260926 14:47:14.544960 13986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:14.546062 13986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:14.547227 13986 hybrid_clock.cc:648] HybridClock initialized: now 1790434034547210 us; error 34 us; skew 500 ppm
Sep 26 14:47:14 dist-test-slave-s7k6 krb5kdc[13973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:14.549921 13986 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:47:14.550880 13986 webserver.cc:460] Webserver started at http://127.0.98.62:36189/ using document root <none> and password file <none>
I20260926 14:47:14.551427 13986 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:14.551487 13986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:14.551718 13986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:14.553795 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "6a7e8eb4743e40408707c4bf3f72d444"
format_stamp: "Formatted at 2026-09-26 14:47:14 on dist-test-slave-s7k6"
server_key: "987c91cfb6c6ba908f1b54e0c5d2ce96"
server_key_iv: "23fded9d52814158bdc5ad01efc1c3d5"
server_key_version: "encryptionkey@0"
I20260926 14:47:14.554360 13986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a7e8eb4743e40408707c4bf3f72d444"
format_stamp: "Formatted at 2026-09-26 14:47:14 on dist-test-slave-s7k6"
server_key: "987c91cfb6c6ba908f1b54e0c5d2ce96"
server_key_iv: "23fded9d52814158bdc5ad01efc1c3d5"
server_key_version: "encryptionkey@0"
I20260926 14:47:14.558235 13986 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 14:47:14.560863 14001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:14.562170 13986 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:47:14.562304 13986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "6a7e8eb4743e40408707c4bf3f72d444"
format_stamp: "Formatted at 2026-09-26 14:47:14 on dist-test-slave-s7k6"
server_key: "987c91cfb6c6ba908f1b54e0c5d2ce96"
server_key_iv: "23fded9d52814158bdc5ad01efc1c3d5"
server_key_version: "encryptionkey@0"
I20260926 14:47:14.562417 13986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:14.582894 13986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:14.586053 13986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:14.586242 13986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:14.593468 13986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42543
I20260926 14:47:14.593477 14053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42543 every 8 connection(s)
I20260926 14:47:14.594534 13986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:47:14.597594 14054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:14.599527 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 13986
I20260926 14:47:14.599647 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:47:14.599948 392 external_mini_cluster.cc:1428] Setting key b256bbe59cec90baa5317ecaeff8e4bc
I20260926 14:47:14.604377 14054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444: Bootstrap starting.
I20260926 14:47:14.606935 14054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:14.607811 14054 log.cc:824] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:47:14 dist-test-slave-s7k6 krb5kdc[13973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:14.610124 14054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444: No bootstrap required, opened a new log
I20260926 14:47:14.614800 14054 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e8eb4743e40408707c4bf3f72d444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42543 } }
I20260926 14:47:14.615095 14054 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:14.615103 14057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:14.601303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.0.98.62:42543)
0926 14:47:14.601727 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:14.601743 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:14.601783 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:47:14.602473 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:14.602501 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:14.602897 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:14.603195 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:14.604022 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:14.605122 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:14.606314 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:14.606840 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:14.606974 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:14.609790 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:14.609795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:14.609810 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:14.609836 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:14.611811 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:14.612258 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:14.612266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:14.612271 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:14.612419 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:14.612792 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:14.612795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:14.612797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:14.613307 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:14.613497 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:14.613754 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 14:47:14.614083 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":118,"threads_started":1}
I20260926 14:47:14.615175 14054 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7e8eb4743e40408707c4bf3f72d444, State: Initialized, Role: FOLLOWER
I20260926 14:47:14.615639 14054 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e8eb4743e40408707c4bf3f72d444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42543 } }
I20260926 14:47:14.615783 14054 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:47:14.615846 14054 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:47:14.615943 14054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:14.616832 14054 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e8eb4743e40408707c4bf3f72d444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42543 } }
I20260926 14:47:14.617187 14054 leader_election.cc:302] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a7e8eb4743e40408707c4bf3f72d444; no voters:
I20260926 14:47:14.617517 14054 leader_election.cc:288] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:47:14.617656 14059 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:47:14.617965 14059 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [term 1 LEADER]: Becoming Leader. State: Replica: 6a7e8eb4743e40408707c4bf3f72d444, State: Running, Role: LEADER
I20260926 14:47:14.618146 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:47:14.618316 14059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e8eb4743e40408707c4bf3f72d444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42543 } }
I20260926 14:47:14.618821 14054 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:47:14.619869 14060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a7e8eb4743e40408707c4bf3f72d444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e8eb4743e40408707c4bf3f72d444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42543 } } }
I20260926 14:47:14.620007 14060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:14.620368 14068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:47:14.620327 14061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a7e8eb4743e40408707c4bf3f72d444. Latest consensus state: current_term: 1 leader_uuid: "6a7e8eb4743e40408707c4bf3f72d444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7e8eb4743e40408707c4bf3f72d444" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42543 } } }
I20260926 14:47:14.620405 14061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444 [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:14.623051 14068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:47:14.628564 14068 catalog_manager.cc:1347] Generated new cluster ID: 8fcefd5d50904c32947262be386bab6c
I20260926 14:47:14.628638 14068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:47:14.643141 14068 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:47:14.644178 14068 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:47:14.653003 14068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a7e8eb4743e40408707c4bf3f72d444: Generated new TSK 0
I20260926 14:47:14.653728 14068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:47:14.673995 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 13986
2026-09-26T14:47:14Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1346 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:47:14 dist-test-slave-s7k6 krb5kdc[14081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:47:14 dist-test-slave-s7k6 krb5kdc[14081](info): set up 2 sockets
Sep 26 14:47:14 dist-test-slave-s7k6 krb5kdc[14081](info): commencing operation
krb5kdc: starting...
W20260926 14:47:15.622814 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:47:15 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:47:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:47:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:47:15.740077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45781
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:15.851171 14094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:15.851574 14094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:15.851648 14094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:15.857410 14094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:47:15.857489 14094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:15.857533 14094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:15.857564 14094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:47:15.857595 14094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:47:15.865458 14094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:15.867264 14094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:15.868728 14094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:47:15.879700 14094 server_base.cc:1034] running on GCE node
W20260926 14:47:15.879657 14102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:15.879901 14099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:15.879657 14100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:15.880726 14094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:15.882068 14094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:15.883239 14094 hybrid_clock.cc:648] HybridClock initialized: now 1790434035883203 us; error 54 us; skew 500 ppm
Sep 26 14:47:15 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:15.886792 14094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:47:15.887774 14094 webserver.cc:460] Webserver started at http://127.0.98.62:38413/ using document root <none> and password file <none>
I20260926 14:47:15.888334 14094 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:15.888399 14094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:15.888605 14094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:15.890736 14094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "71fbb757681f441fb1aeab7bbebddbc0"
format_stamp: "Formatted at 2026-09-26 14:47:15 on dist-test-slave-s7k6"
server_key: "d50dd0c3b4820eb1c40350f334221b43"
server_key_iv: "b95859022b0524837e8179ddc4592b7e"
server_key_version: "encryptionkey@0"
I20260926 14:47:15.891340 14094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "71fbb757681f441fb1aeab7bbebddbc0"
format_stamp: "Formatted at 2026-09-26 14:47:15 on dist-test-slave-s7k6"
server_key: "d50dd0c3b4820eb1c40350f334221b43"
server_key_iv: "b95859022b0524837e8179ddc4592b7e"
server_key_version: "encryptionkey@0"
I20260926 14:47:15.895043 14094 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:47:15.897385 14109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:15.898618 14094 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:47:15.898744 14094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "71fbb757681f441fb1aeab7bbebddbc0"
format_stamp: "Formatted at 2026-09-26 14:47:15 on dist-test-slave-s7k6"
server_key: "d50dd0c3b4820eb1c40350f334221b43"
server_key_iv: "b95859022b0524837e8179ddc4592b7e"
server_key_version: "encryptionkey@0"
I20260926 14:47:15.898891 14094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:15.917181 14094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:15.920269 14094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:15.920449 14094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:15.927598 14094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45781
I20260926 14:47:15.927879 14161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45781 every 8 connection(s)
I20260926 14:47:15.928668 14094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:47:15.931463 14162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:15.936180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 14094
I20260926 14:47:15.936293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:47:15.936546 392 external_mini_cluster.cc:1428] Setting key ff27fae99ea8249bee297ad91e083169
I20260926 14:47:15.937341 14162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Bootstrap starting.
I20260926 14:47:15.940058 14162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:15.940893 14162 log.cc:824] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:15.942907 14162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: No bootstrap required, opened a new log
Sep 26 14:47:15 dist-test-slave-s7k6 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:15.946401 14162 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:15.946707 14162 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:15.946789 14162 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71fbb757681f441fb1aeab7bbebddbc0, State: Initialized, Role: FOLLOWER
I20260926 14:47:15.947280 14162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:15.947434 14162 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:47:15.947502 14162 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:47:15.947603 14162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:15.948454 14162 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:15.948788 14162 leader_election.cc:302] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71fbb757681f441fb1aeab7bbebddbc0; no voters:
I20260926 14:47:15.949087 14162 leader_election.cc:288] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:47:15.949165 14167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:47:15.949348 14167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 1 LEADER]: Becoming Leader. State: Replica: 71fbb757681f441fb1aeab7bbebddbc0, State: Running, Role: LEADER
I20260926 14:47:15.949652 14167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:15.950414 14162 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:47:15.951252 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:15.937959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.0.98.62:45781)
0926 14:47:15.938401 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:15.938423 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:15.938471 (+ 48us) server_negotiation.cc:407] Connection header received
0926 14:47:15.939321 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:15.939344 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:15.939758 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:15.940126 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:15.940834 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:15.942169 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:15.943080 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:15.943576 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:15.943727 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:15.946104 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:15.946109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:15.946123 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:15.946151 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:15.947911 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:15.948334 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:15.948340 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:15.948346 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:15.948458 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:15.948728 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:15.948731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:15.948733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:15.949171 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:15.949347 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:15.950058 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 14:47:15.950306 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":132,"threads_started":1}
I20260926 14:47:15.951328 14168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71fbb757681f441fb1aeab7bbebddbc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } } }
I20260926 14:47:15.951458 14168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:15.951669 14168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71fbb757681f441fb1aeab7bbebddbc0. Latest consensus state: current_term: 1 leader_uuid: "71fbb757681f441fb1aeab7bbebddbc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } } }
I20260926 14:47:15.951756 14168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: This master's current role is: LEADER
W20260926 14:47:15.955716 14182 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:47:15.955809 14182 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:47:15.955878 14176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:47:15.956808 14176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:47:15.962901 14176 catalog_manager.cc:1347] Generated new cluster ID: 0ca8a5c109c849bbafed755ef2b3149e
I20260926 14:47:15.962983 14176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:47:15.984009 14176 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:47:15.985002 14176 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:47:15.999588 14176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Generated new TSK 0
I20260926 14:47:16.000401 14176 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:47:16.027571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45781
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:16.139791 14188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:16.140089 14188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:16.140158 14188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:16.144275 14188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:16.144339 14188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:16.144420 14188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:47:16.149762 14188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:16.150884 14188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:16.151944 14188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:16.162138 14193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:16.162303 14188 server_base.cc:1034] running on GCE node
W20260926 14:47:16.162129 14196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:16.162138 14194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:16.162875 14188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:16.163563 14188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:16.164734 14188 hybrid_clock.cc:648] HybridClock initialized: now 1790434036164708 us; error 40 us; skew 500 ppm
Sep 26 14:47:16 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:16.167675 14188 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:47:16.168668 14188 webserver.cc:460] Webserver started at http://127.0.98.1:37045/ using document root <none> and password file <none>
I20260926 14:47:16.169247 14188 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:16.169315 14188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:16.169514 14188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:16.171667 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "dd97849388ff46e8a6fe07a1c975df91"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "390988d1e209ae5d9f5319854e7e52aa"
server_key_iv: "6b98c6a7312340f006a9e2599ebbc56d"
server_key_version: "encryptionkey@0"
I20260926 14:47:16.172317 14188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dd97849388ff46e8a6fe07a1c975df91"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "390988d1e209ae5d9f5319854e7e52aa"
server_key_iv: "6b98c6a7312340f006a9e2599ebbc56d"
server_key_version: "encryptionkey@0"
I20260926 14:47:16.176244 14188 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:47:16.178870 14203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:16.180220 14188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:47:16.180326 14188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "dd97849388ff46e8a6fe07a1c975df91"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "390988d1e209ae5d9f5319854e7e52aa"
server_key_iv: "6b98c6a7312340f006a9e2599ebbc56d"
server_key_version: "encryptionkey@0"
I20260926 14:47:16.180419 14188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:16.194901 14188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:16.198081 14188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:16.198251 14188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:16.198836 14188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:16.199661 14188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:16.199707 14188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:16.199745 14188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:16.199765 14188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:16.208295 14188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41871
I20260926 14:47:16.208324 14316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41871 every 8 connection(s)
I20260926 14:47:16.209163 14188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:47:16.214092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 14188
I20260926 14:47:16.214193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:47:16.214442 392 external_mini_cluster.cc:1428] Setting key 1323a2fbc8238477b57933af64547880
Sep 26 14:47:16 dist-test-slave-s7k6 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 14:47:16.223049 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.211102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36417 (local address 127.0.98.62:45781)
0926 14:47:16.211253 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:16.211256 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:16.212108 (+ 852us) server_negotiation.cc:407] Connection header received
0926 14:47:16.213415 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:16.213419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:16.213475 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:16.213551 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:16.214827 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.215453 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.216597 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.216839 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.216926 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.219388 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:16.219391 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:16.219393 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:16.219417 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:16.221059 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:16.221547 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:16.221550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:16.221552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:16.221637 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:16.221945 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:16.221948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:16.221950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:16.222141 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:16.222223 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:16.222715 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 14:47:16.222870 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 14:47:16.223902 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.211439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:36417)
0926 14:47:16.211927 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:16.211973 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:16.213207 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:16.213734 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:16.213741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:16.214096 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:16.214662 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.214678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.215593 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.215596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.216476 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.216487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.216986 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.216991 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.217151 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.217759 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:16.217776 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:16.219220 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:16.221185 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:16.221191 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:16.221207 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:16.221453 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:16.221719 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:16.221722 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:16.221724 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:16.221861 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:16.222237 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:16.222244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:16.222495 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 14:47:16.222660 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 14:47:16.223021 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":118,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 14:47:16.225131 14317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45781
I20260926 14:47:16.225360 14317 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:16.225847 14317 heartbeater.cc:511] Master 127.0.98.62:45781 requested a full tablet report, sending...
I20260926 14:47:16.227466 14126 ts_manager.cc:194] Registered new tserver with Master: dd97849388ff46e8a6fe07a1c975df91 (127.0.98.1:41871)
I20260926 14:47:16.228590 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:47:16.241766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45781
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:16.352653 14322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:16.352928 14322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:16.352972 14322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:16.357079 14322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:16.357141 14322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:16.357213 14322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:47:16.362828 14322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:16.363973 14322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:16.365266 14322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:16.375141 14328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:16.375141 14327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:16.375324 14322 server_base.cc:1034] running on GCE node
W20260926 14:47:16.375211 14330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:16.375820 14322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:16.376461 14322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:16.377650 14322 hybrid_clock.cc:648] HybridClock initialized: now 1790434036377629 us; error 36 us; skew 500 ppm
Sep 26 14:47:16 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:16.380393 14322 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:47:16.381248 14322 webserver.cc:460] Webserver started at http://127.0.98.2:44969/ using document root <none> and password file <none>
I20260926 14:47:16.381781 14322 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:16.381847 14322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:16.382033 14322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:16.384145 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "747dcdb586f74a30b7ac99f74601971c"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "9ad4b51d360ca1b806bea6b6d3e50ea5"
server_key_iv: "825829a19defeef7de13609a13e66955"
server_key_version: "encryptionkey@0"
I20260926 14:47:16.384727 14322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "747dcdb586f74a30b7ac99f74601971c"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "9ad4b51d360ca1b806bea6b6d3e50ea5"
server_key_iv: "825829a19defeef7de13609a13e66955"
server_key_version: "encryptionkey@0"
I20260926 14:47:16.388387 14322 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:47:16.390848 14337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:16.392086 14322 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:47:16.392201 14322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "747dcdb586f74a30b7ac99f74601971c"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "9ad4b51d360ca1b806bea6b6d3e50ea5"
server_key_iv: "825829a19defeef7de13609a13e66955"
server_key_version: "encryptionkey@0"
I20260926 14:47:16.392328 14322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:16.420603 14322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:16.423427 14322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:16.423609 14322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:16.424201 14322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:16.425043 14322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:16.425091 14322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:16.425143 14322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:16.425171 14322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:16.433483 14322 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40219
I20260926 14:47:16.433516 14450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40219 every 8 connection(s)
I20260926 14:47:16.434432 14322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:47:16.438076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 14322
I20260926 14:47:16.438184 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:47:16.438438 392 external_mini_cluster.cc:1428] Setting key b0fe9f371c268b922c948c9cf9cf248f
Sep 26 14:47:16 dist-test-slave-s7k6 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 14:47:16.448630 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.436337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40275 (local address 127.0.98.62:45781)
0926 14:47:16.436488 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:16.436493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:16.437607 (+ 1114us) server_negotiation.cc:407] Connection header received
0926 14:47:16.438648 (+ 1041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:16.438652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:16.438711 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:16.438849 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:16.440263 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.441023 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.442312 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.442642 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.442732 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.445219 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:16.445222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:16.445224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:16.445248 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:16.446549 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:16.447111 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:16.447114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:16.447116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:16.447192 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:16.447588 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:16.447591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:16.447593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:16.447791 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:16.447873 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:16.448380 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 14:47:16.448501 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:47:16.449476 14453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.436751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:40275)
0926 14:47:16.437433 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:16.437485 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:16.438428 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:16.439050 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:16.439061 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:16.439452 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:16.440098 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.440114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.441163 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.441166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.442179 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.442190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.442807 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.442811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.442958 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.443529 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:16.443545 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:16.445048 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:16.446667 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:16.446673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:16.446691 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:16.447007 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:16.447301 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:16.447305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:16.447307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:16.447500 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:16.447875 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:16.447880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:16.448125 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 14:47:16.448325 (+ 200us) client_negotiation.cc:240] Negotiation successful
0926 14:47:16.448605 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:47:16.450783 14451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45781
I20260926 14:47:16.451054 14451 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:16.451639 14451 heartbeater.cc:511] Master 127.0.98.62:45781 requested a full tablet report, sending...
I20260926 14:47:16.452694 14126 ts_manager.cc:194] Registered new tserver with Master: 747dcdb586f74a30b7ac99f74601971c (127.0.98.2:40219)
I20260926 14:47:16.453258 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:47:16.465412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45781
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:16.576021 14456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:16.576306 14456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:16.576359 14456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:16.580652 14456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:16.580724 14456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:16.580785 14456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:47:16.586220 14456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:16.587316 14456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:16.588330 14456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:16.598076 14461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:16.598085 14462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:16.598239 14456 server_base.cc:1034] running on GCE node
W20260926 14:47:16.598398 14464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:16.598804 14456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:16.599444 14456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:16.600615 14456 hybrid_clock.cc:648] HybridClock initialized: now 1790434036600591 us; error 32 us; skew 500 ppm
Sep 26 14:47:16 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:16.603576 14456 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:47:16.604471 14456 webserver.cc:460] Webserver started at http://127.0.98.3:45361/ using document root <none> and password file <none>
I20260926 14:47:16.605031 14456 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:16.605098 14456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:16.605299 14456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:16.607456 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "f1b9cf19ecf0a1883c6437091c651c75"
server_key_iv: "5a64b7e89ec0a139a225bab11a6c7076"
server_key_version: "encryptionkey@0"
I20260926 14:47:16.607995 14456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "f1b9cf19ecf0a1883c6437091c651c75"
server_key_iv: "5a64b7e89ec0a139a225bab11a6c7076"
server_key_version: "encryptionkey@0"
I20260926 14:47:16.611809 14456 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:47:16.614217 14471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:16.615495 14456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:47:16.615607 14456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "f1b9cf19ecf0a1883c6437091c651c75"
server_key_iv: "5a64b7e89ec0a139a225bab11a6c7076"
server_key_version: "encryptionkey@0"
I20260926 14:47:16.615712 14456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:16.641343 14456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:16.644852 14456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:16.645045 14456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:16.645711 14456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:16.646569 14456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:16.646616 14456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:16.646670 14456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:16.646698 14456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:16.655627 14456 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38117
I20260926 14:47:16.655658 14584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38117 every 8 connection(s)
I20260926 14:47:16.656540 14456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:47:16.661593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 14456
I20260926 14:47:16.661695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:47:16.662001 392 external_mini_cluster.cc:1428] Setting key db93e533c6da8ba2164e1d23364f365f
Sep 26 14:47:16 dist-test-slave-s7k6 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:16.669749 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.658382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38009 (local address 127.0.98.62:45781)
0926 14:47:16.658557 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:16.658560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:16.659684 (+ 1124us) server_negotiation.cc:407] Connection header received
0926 14:47:16.660566 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:16.660570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:16.660625 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:16.660704 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:16.661953 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.662613 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.663960 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.664179 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.664264 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.666664 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:16.666667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:16.666669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:16.666695 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:16.667920 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:16.668389 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:16.668391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:16.668392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:16.668466 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:16.668723 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:16.668725 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:16.668726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:16.668922 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:16.668996 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:16.669510 (+ 514us) server_negotiation.cc:299] Negotiation successful
0926 14:47:16.669618 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 14:47:16.670589 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.658814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:38009)
0926 14:47:16.659511 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:16.659553 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:16.660363 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:16.660888 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:16.660895 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:16.661275 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:16.661805 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.661821 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.662731 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.662735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.663837 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.663848 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.664335 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.664339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.664506 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.665042 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:16.665058 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:16.666498 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:16.668038 (+ 1540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:16.668043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:16.668056 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:16.668312 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:16.668532 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:16.668534 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:16.668535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:16.668662 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:16.669008 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:16.669014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:16.669270 (+ 256us) client_negotiation.cc:769] Sending connection context
0926 14:47:16.669445 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 14:47:16.669709 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":118,"threads_started":1}
I20260926 14:47:16.671747 14585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45781
I20260926 14:47:16.672026 14585 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:16.672559 14585 heartbeater.cc:511] Master 127.0.98.62:45781 requested a full tablet report, sending...
I20260926 14:47:16.673684 14126 ts_manager.cc:194] Registered new tserver with Master: 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117)
I20260926 14:47:16.674286 14126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38009
I20260926 14:47:16.676105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:47:16.685230 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.678134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42406 (local address 127.0.98.62:45781)
0926 14:47:16.678280 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:16.678284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:16.678434 (+ 150us) server_negotiation.cc:407] Connection header received
0926 14:47:16.678526 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:16.678528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:16.678568 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:16.678636 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:16.679434 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.680003 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.680881 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.681099 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.681185 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.682501 (+ 1316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:16.682504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:16.682506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:16.682529 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:16.683712 (+ 1183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:16.684183 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:16.684185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:16.684187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:16.684266 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:16.684484 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:16.684486 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:16.684487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:16.684683 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:16.684814 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:16.684967 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 14:47:16.685093 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:47:16.688182 14126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:47:16.691322 14126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:47:16.705487 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.699738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38117 (local address 127.0.0.1:38056)
0926 14:47:16.700531 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:16.700544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:16.700675 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:16.701429 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:16.701434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:16.701457 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:16.701644 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.701652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.702739 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.702742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.704020 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.704029 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.704931 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.704935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.705043 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.705082 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 14:47:16.705203 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 14:47:16.705341 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":104,"threads_started":1}
I20260926 14:47:16.705495 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.700457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38056 (local address 127.0.98.3:38117)
0926 14:47:16.700904 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:16.700918 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:16.700941 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:47:16.701016 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:16.701022 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:16.701239 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:16.701405 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:16.701764 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.702620 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.704130 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.704808 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.704923 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.705127 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:16.705234 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 14:47:16.705339 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":141,"threads_started":1}
I20260926 14:47:16.706240 14596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.699642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40219 (local address 127.0.0.1:34718)
0926 14:47:16.700226 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:16.700275 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:16.700732 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:16.701763 (+ 1031us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:16.701766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:16.701806 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:16.701905 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.701910 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.703461 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.703464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.704351 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.704360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.705542 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.705546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.705939 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.705960 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 14:47:16.706034 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 14:47:16.706103 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":43,"spinlock_wait_cycles":896,"thread_start_us":84,"threads_started":1}
I20260926 14:47:16.706321 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.699778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.0.98.2:40219)
0926 14:47:16.701192 (+ 1414us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:16.701202 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:16.701223 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:47:16.701292 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:16.701297 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:16.701468 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:16.701600 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:16.702013 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.703292 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.704470 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.705259 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.705352 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.705479 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:16.706083 (+ 604us) server_negotiation.cc:299] Negotiation successful
0926 14:47:16.706182 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1317,"thread_start_us":87,"threads_started":1}
I20260926 14:47:16.706830 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.699946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41871 (local address 127.0.0.1:33198)
0926 14:47:16.700886 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:16.700898 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:16.700967 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:16.701731 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:16.701734 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:16.701751 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:16.701847 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.701851 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.703491 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.703494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.705158 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.705166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.706339 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.706343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.706429 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.706451 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:47:16.706515 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:47:16.706654 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":824,"spinlock_wait_cycles":12416,"thread_start_us":79,"threads_started":1}
I20260926 14:47:16.707322 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.700447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.0.98.1:41871)
0926 14:47:16.701093 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:16.701106 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:16.701125 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:47:16.701185 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:16.701189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:16.701367 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:16.701546 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:16.702017 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.703292 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.705280 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.706060 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.706155 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.706280 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:16.707088 (+ 808us) server_negotiation.cc:299] Negotiation successful
0926 14:47:16.707185 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":86,"threads_started":1}
I20260926 14:47:16.708767 14519 tablet_service.cc:1467] Processing CreateTablet for tablet 63aeb89161ec4f6fb6af93fc68cb421a (DEFAULT_TABLE table=test-table [id=870ac701c477473c9254f13f493bdd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:16.709160 14385 tablet_service.cc:1467] Processing CreateTablet for tablet 63aeb89161ec4f6fb6af93fc68cb421a (DEFAULT_TABLE table=test-table [id=870ac701c477473c9254f13f493bdd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:16.709733 14519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63aeb89161ec4f6fb6af93fc68cb421a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:16.710218 14385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63aeb89161ec4f6fb6af93fc68cb421a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:16.709826 14251 tablet_service.cc:1467] Processing CreateTablet for tablet 63aeb89161ec4f6fb6af93fc68cb421a (DEFAULT_TABLE table=test-table [id=870ac701c477473c9254f13f493bdd19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:16.710884 14251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63aeb89161ec4f6fb6af93fc68cb421a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:16.716212 14602 tablet_bootstrap.cc:492] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Bootstrap starting.
I20260926 14:47:16.717901 14603 tablet_bootstrap.cc:492] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c: Bootstrap starting.
I20260926 14:47:16.718547 14602 tablet_bootstrap.cc:654] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:16.719074 14604 tablet_bootstrap.cc:492] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Bootstrap starting.
I20260926 14:47:16.719906 14602 log.cc:824] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:16.720484 14603 tablet_bootstrap.cc:654] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:16.721297 14604 tablet_bootstrap.cc:654] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:16.721305 14603 log.cc:824] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:16.721727 14602 tablet_bootstrap.cc:492] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb: No bootstrap required, opened a new log
I20260926 14:47:16.721943 14602 ts_tablet_manager.cc:1397] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 14:47:16.722461 14604 log.cc:824] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:16.723289 14603 tablet_bootstrap.cc:492] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c: No bootstrap required, opened a new log
I20260926 14:47:16.723485 14603 ts_tablet_manager.cc:1397] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:47:16.724725 14604 tablet_bootstrap.cc:492] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: No bootstrap required, opened a new log
I20260926 14:47:16.724979 14604 ts_tablet_manager.cc:1397] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260926 14:47:16.725500 14602 raft_consensus.cc:348] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:16.725751 14602 raft_consensus.cc:374] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:16.725803 14602 raft_consensus.cc:729] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b59a31f1cef4b1691b5f40f3fa2d9eb, State: Initialized, Role: FOLLOWER
I20260926 14:47:16.726226 14602 consensus_queue.cc:260] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:16.726894 14585 heartbeater.cc:503] Master 127.0.98.62:45781 was elected leader, sending a full tablet report...
I20260926 14:47:16.727327 14603 raft_consensus.cc:348] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:16.727613 14603 raft_consensus.cc:374] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:16.727667 14603 raft_consensus.cc:729] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747dcdb586f74a30b7ac99f74601971c, State: Initialized, Role: FOLLOWER
I20260926 14:47:16.728157 14603 consensus_queue.cc:260] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:16.728359 14602 ts_tablet_manager.cc:1428] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260926 14:47:16.728785 14604 raft_consensus.cc:348] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:16.729027 14604 raft_consensus.cc:374] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:16.729074 14604 raft_consensus.cc:729] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd97849388ff46e8a6fe07a1c975df91, State: Initialized, Role: FOLLOWER
I20260926 14:47:16.729084 14451 heartbeater.cc:503] Master 127.0.98.62:45781 was elected leader, sending a full tablet report...
I20260926 14:47:16.729159 14603 ts_tablet_manager.cc:1428] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:47:16.729504 14604 consensus_queue.cc:260] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:16.730029 14317 heartbeater.cc:503] Master 127.0.98.62:45781 was elected leader, sending a full tablet report...
I20260926 14:47:16.730345 14604 ts_tablet_manager.cc:1428] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:47:16.762820 14610 raft_consensus.cc:482] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:47:16.763023 14610 raft_consensus.cc:504] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:16.764219 14610 leader_election.cc:288] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 747dcdb586f74a30b7ac99f74601971c (127.0.98.2:40219), 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117)
I20260926 14:47:16.768136 14598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.764541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45929 (local address 127.0.98.2:40219)
0926 14:47:16.764664 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:16.764668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:16.764901 (+ 233us) server_negotiation.cc:407] Connection header received
0926 14:47:16.765064 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:16.765066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:16.765112 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:16.765179 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:16.765461 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.765953 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.767166 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.767578 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.767677 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.767737 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:16.767935 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 14:47:16.768012 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:47:16.768142 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.764583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41443 (local address 127.0.98.3:38117)
0926 14:47:16.764788 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:16.764792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:16.764907 (+ 115us) server_negotiation.cc:407] Connection header received
0926 14:47:16.765068 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:16.765071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:16.765114 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:16.765176 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:16.765451 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.766013 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.767164 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.767569 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.767659 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.767737 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:16.767932 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 14:47:16.768008 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260926 14:47:16.768186 14611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.764444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40219 (local address 127.0.98.1:45929)
0926 14:47:16.764785 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:16.764800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:16.764952 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:16.765221 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:16.765224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:16.765241 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:16.765350 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.765354 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.766058 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.766061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.767038 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.767049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.767702 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.767706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.767818 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.767835 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:47:16.767902 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:47:16.767980 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":28,"spinlock_wait_cycles":38144,"thread_start_us":125,"threads_started":1}
I20260926 14:47:16.768191 14612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.764508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38117 (local address 127.0.98.1:41443)
0926 14:47:16.764805 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:16.764819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:16.764953 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:16.765197 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:16.765200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:16.765215 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:16.765342 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.765348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.766120 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.766122 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.767038 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:16.767049 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.767700 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.767703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:16.767810 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.767829 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:47:16.767894 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:47:16.767974 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":83,"spinlock_wait_cycles":10880,"thread_start_us":121,"threads_started":1}
I20260926 14:47:16.769047 14405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63aeb89161ec4f6fb6af93fc68cb421a" candidate_uuid: "dd97849388ff46e8a6fe07a1c975df91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747dcdb586f74a30b7ac99f74601971c" is_pre_election: true
I20260926 14:47:16.769050 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63aeb89161ec4f6fb6af93fc68cb421a" candidate_uuid: "dd97849388ff46e8a6fe07a1c975df91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" is_pre_election: true
I20260926 14:47:16.769375 14405 raft_consensus.cc:2454] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd97849388ff46e8a6fe07a1c975df91 in term 0.
I20260926 14:47:16.769375 14539 raft_consensus.cc:2454] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd97849388ff46e8a6fe07a1c975df91 in term 0.
I20260926 14:47:16.769871 14206 leader_election.cc:302] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b59a31f1cef4b1691b5f40f3fa2d9eb, dd97849388ff46e8a6fe07a1c975df91; no voters:
I20260926 14:47:16.770167 14610 raft_consensus.cc:2781] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:47:16.770277 14610 raft_consensus.cc:482] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:47:16.770308 14610 raft_consensus.cc:3037] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:16.771518 14610 raft_consensus.cc:504] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:16.771863 14610 leader_election.cc:288] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 1 election: Requested vote from peers 747dcdb586f74a30b7ac99f74601971c (127.0.98.2:40219), 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117)
I20260926 14:47:16.772233 14405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63aeb89161ec4f6fb6af93fc68cb421a" candidate_uuid: "dd97849388ff46e8a6fe07a1c975df91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747dcdb586f74a30b7ac99f74601971c"
I20260926 14:47:16.772236 14539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63aeb89161ec4f6fb6af93fc68cb421a" candidate_uuid: "dd97849388ff46e8a6fe07a1c975df91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb"
I20260926 14:47:16.772351 14539 raft_consensus.cc:3037] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:16.772354 14405 raft_consensus.cc:3037] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:16.773382 14539 raft_consensus.cc:2454] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd97849388ff46e8a6fe07a1c975df91 in term 1.
I20260926 14:47:16.773386 14405 raft_consensus.cc:2454] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd97849388ff46e8a6fe07a1c975df91 in term 1.
I20260926 14:47:16.773665 14207 leader_election.cc:302] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747dcdb586f74a30b7ac99f74601971c, dd97849388ff46e8a6fe07a1c975df91; no voters:
I20260926 14:47:16.773898 14610 raft_consensus.cc:2781] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:47:16.774106 14610 raft_consensus.cc:686] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 1 LEADER]: Becoming Leader. State: Replica: dd97849388ff46e8a6fe07a1c975df91, State: Running, Role: LEADER
I20260926 14:47:16.774482 14610 consensus_queue.cc:237] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:16.777782 14124 catalog_manager.cc:5495] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd97849388ff46e8a6fe07a1c975df91 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dd97849388ff46e8a6fe07a1c975df91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } health_report { overall_health: HEALTHY } } }
I20260926 14:47:16.793025 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.789588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.0.98.1:41871)
0926 14:47:16.789714 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:16.789719 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:16.789734 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:47:16.789887 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:16.789890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:16.789945 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:16.790031 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:47:16.790282 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.790803 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.791621 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:16.791835 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:16.791920 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:16.792207 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:47:16.792645 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:47:16.792755 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:16.792827 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 14:47:16.792887 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:47:16.799037 14405 raft_consensus.cc:1261] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Refusing update from remote peer dd97849388ff46e8a6fe07a1c975df91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:16.799037 14539 raft_consensus.cc:1261] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 1 FOLLOWER]: Refusing update from remote peer dd97849388ff46e8a6fe07a1c975df91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:16.799732 14610 consensus_queue.cc:1035] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:47:16.799935 14613 consensus_queue.cc:1035] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:47:16.808564 14618 mvcc.cc:205] Tried to move back new op lower bound from 7333617814721466368 to 7333617814631108608. Current Snapshot: MvccSnapshot[applied={T|T < 7333617814721466368}]
I20260926 14:47:16.808737 14617 mvcc.cc:205] Tried to move back new op lower bound from 7333617814721466368 to 7333617814631108608. Current Snapshot: MvccSnapshot[applied={T|T < 7333617814721466368}]
I20260926 14:47:16.816887 14124 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:47:16.817049 14124 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:47:16.819370 14124 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Sending DeleteTablet for 3 replicas of tablet 63aeb89161ec4f6fb6af93fc68cb421a
I20260926 14:47:16.820176 14519 tablet_service.cc:1514] Processing DeleteTablet for tablet 63aeb89161ec4f6fb6af93fc68cb421a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38056
I20260926 14:47:16.820168 14385 tablet_service.cc:1514] Processing DeleteTablet for tablet 63aeb89161ec4f6fb6af93fc68cb421a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34718
I20260926 14:47:16.820262 14251 tablet_service.cc:1514] Processing DeleteTablet for tablet 63aeb89161ec4f6fb6af93fc68cb421a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33198
I20260926 14:47:16.820700 14626 tablet_replica.cc:330] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb: stopping tablet replica
I20260926 14:47:16.820870 14627 tablet_replica.cc:330] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c: stopping tablet replica
I20260926 14:47:16.820904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 14188
I20260926 14:47:16.821094 14627 raft_consensus.cc:2229] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:47:16.821350 14627 raft_consensus.cc:2258] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:16.821844 14626 raft_consensus.cc:2229] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:47:16.822080 14626 raft_consensus.cc:2258] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:16.823758 14626 ts_tablet_manager.cc:1905] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:16.823794 14627 ts_tablet_manager.cc:1905] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:16.826030 14626 ts_tablet_manager.cc:1918] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:47:16.826126 14626 log.cc:1196] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/63aeb89161ec4f6fb6af93fc68cb421a
I20260926 14:47:16.826339 14627 ts_tablet_manager.cc:1918] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:47:16.826432 14627 log.cc:1196] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/63aeb89161ec4f6fb6af93fc68cb421a
I20260926 14:47:16.826437 14626 ts_tablet_manager.cc:1939] T 63aeb89161ec4f6fb6af93fc68cb421a P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Deleting consensus metadata
I20260926 14:47:16.826722 14627 ts_tablet_manager.cc:1939] T 63aeb89161ec4f6fb6af93fc68cb421a P 747dcdb586f74a30b7ac99f74601971c: Deleting consensus metadata
I20260926 14:47:16.827630 14112 catalog_manager.cc:4841] TS 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117): tablet 63aeb89161ec4f6fb6af93fc68cb421a (table test-table [id=870ac701c477473c9254f13f493bdd19]) successfully deleted
I20260926 14:47:16.827831 14113 catalog_manager.cc:4841] TS 747dcdb586f74a30b7ac99f74601971c (127.0.98.2:40219): tablet 63aeb89161ec4f6fb6af93fc68cb421a (table test-table [id=870ac701c477473c9254f13f493bdd19]) successfully deleted
W20260926 14:47:16.828295 14110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41871 (error 108)
I20260926 14:47:16.828486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 14322
I20260926 14:47:16.829628 14599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:16.829132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41871 (local address 127.0.0.1:33208)
0926 14:47:16.829491 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:16.829553 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278}
W20260926 14:47:16.829831 14110 catalog_manager.cc:4584] TS dd97849388ff46e8a6fe07a1c975df91 (127.0.98.1:41871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63aeb89161ec4f6fb6af93fc68cb421a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41871: connect: Connection refused (error 111)
I20260926 14:47:16.834110 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 14456
I20260926 14:47:16.839742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 14094
I20260926 14:47:16.845391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45781
--webserver_interface=127.0.98.62
--webserver_port=38413
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:16.955780 14630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:16.956096 14630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:16.956139 14630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:16.960292 14630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:47:16.960367 14630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:16.960388 14630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:16.960404 14630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:47:16.960418 14630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:47:16.966078 14630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:16.967284 14630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:16.968295 14630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:16.981329 14639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:16.981357 14636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:16.981333 14637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:16.981838 14630 server_base.cc:1034] running on GCE node
I20260926 14:47:16.982474 14630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:16.983760 14630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:16.984954 14630 hybrid_clock.cc:648] HybridClock initialized: now 1790434036984928 us; error 43 us; skew 500 ppm
Sep 26 14:47:16 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:16.987905 14630 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:47:16.988839 14630 webserver.cc:460] Webserver started at http://127.0.98.62:38413/ using document root <none> and password file <none>
I20260926 14:47:16.989391 14630 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:16.989464 14630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:16.993683 14630 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:47:16.996137 14646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:16.997581 14630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:47:16.997713 14630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "71fbb757681f441fb1aeab7bbebddbc0"
format_stamp: "Formatted at 2026-09-26 14:47:15 on dist-test-slave-s7k6"
server_key: "d50dd0c3b4820eb1c40350f334221b43"
server_key_iv: "b95859022b0524837e8179ddc4592b7e"
server_key_version: "encryptionkey@0"
I20260926 14:47:16.998397 14630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:17.027688 14630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:17.030598 14630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:17.030799 14630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:17.037773 14630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45781
I20260926 14:47:17.038039 14698 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45781 every 8 connection(s)
I20260926 14:47:17.038944 14630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:47:17.041787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 14630
I20260926 14:47:17.042202 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41871
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45781
--webserver_port=37045
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:17.043678 14699 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:47:17.045910 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Bootstrap starting.
I20260926 14:47:17.069047 14699 log.cc:824] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:17.080677 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:47:17.081146 14699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Bootstrap complete.
I20260926 14:47:17.086776 14699 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:17.087327 14699 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71fbb757681f441fb1aeab7bbebddbc0, State: Initialized, Role: FOLLOWER
I20260926 14:47:17.087908 14699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:17.088063 14699 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:47:17.088140 14699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:47:17.088238 14699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:47:17.090857 14699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:17.091226 14699 leader_election.cc:302] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71fbb757681f441fb1aeab7bbebddbc0; no voters:
I20260926 14:47:17.091555 14699 leader_election.cc:288] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:47:17.091687 14703 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:47:17.091928 14703 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 2 LEADER]: Becoming Leader. State: Replica: 71fbb757681f441fb1aeab7bbebddbc0, State: Running, Role: LEADER
I20260926 14:47:17.092239 14703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:17.092799 14699 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:47:17.093672 14704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "71fbb757681f441fb1aeab7bbebddbc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } } }
I20260926 14:47:17.093825 14704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:17.094460 14705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71fbb757681f441fb1aeab7bbebddbc0. Latest consensus state: current_term: 2 leader_uuid: "71fbb757681f441fb1aeab7bbebddbc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } } }
I20260926 14:47:17.094614 14705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:17.097863 14719 catalog_manager.cc:1259] Loaded cluster ID: 0ca8a5c109c849bbafed755ef2b3149e
I20260926 14:47:17.097929 14719 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: loading cluster ID for follower catalog manager: success
W20260926 14:47:17.098935 14719 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260926 14:47:17.099009 14719 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260926 14:47:17.099069 14712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:47:17.101488 14712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:47:17.101980 14712 catalog_manager.cc:1259] Loaded cluster ID: 0ca8a5c109c849bbafed755ef2b3149e
I20260926 14:47:17.102038 14712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:47:17.103642 14712 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 14:47:17.107997 14712 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 14:47:17.113569 14712 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:47:17.114415 14712 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Loaded TSK: 0
I20260926 14:47:17.115753 14712 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:47:17.178730 14701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:17.179031 14701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:17.179080 14701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:17.183195 14701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:17.183265 14701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:17.183337 14701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:47:17.188755 14701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37045
--tserver_master_addrs=127.0.98.62:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:17.189815 14701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:17.190876 14701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:17.200917 14727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:17.200920 14729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:17.200932 14726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:17.201346 14701 server_base.cc:1034] running on GCE node
I20260926 14:47:17.201794 14701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:17.202484 14701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:17.203657 14701 hybrid_clock.cc:648] HybridClock initialized: now 1790434037203645 us; error 71 us; skew 500 ppm
Sep 26 14:47:17 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:17.206496 14701 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:47:17.207494 14701 webserver.cc:460] Webserver started at http://127.0.98.1:37045/ using document root <none> and password file <none>
I20260926 14:47:17.208038 14701 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:17.208118 14701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:17.212023 14701 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 14:47:17.214138 14736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:17.215433 14701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:47:17.215544 14701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "dd97849388ff46e8a6fe07a1c975df91"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "390988d1e209ae5d9f5319854e7e52aa"
server_key_iv: "6b98c6a7312340f006a9e2599ebbc56d"
server_key_version: "encryptionkey@0"
I20260926 14:47:17.216158 14701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:17.240576 14701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:17.243922 14701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:17.244100 14701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:17.244697 14701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:17.245841 14743 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:47:17.250144 14701 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:47:17.250213 14701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 14:47:17.250267 14701 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:47:17.252067 14701 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:47:17.252115 14701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260926 14:47:17.252398 14743 tablet_bootstrap.cc:492] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Bootstrap starting.
I20260926 14:47:17.261152 14701 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41871
I20260926 14:47:17.261264 14850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41871 every 8 connection(s)
I20260926 14:47:17.262171 14701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:47:17.269636 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 14701
I20260926 14:47:17.270118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40219
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45781
--webserver_port=44969
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:17.271273 14743 log.cc:824] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:47:17 dist-test-slave-s7k6 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:17.280637 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.264226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45441 (local address 127.0.98.62:45781)
0926 14:47:17.264579 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:17.264597 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:17.265011 (+ 414us) server_negotiation.cc:407] Connection header received
0926 14:47:17.266152 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:17.266163 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:17.266480 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:17.266827 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:17.268008 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.269015 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.270552 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.271285 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.272063 (+ 778us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.274378 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:17.274385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:17.274405 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:17.274443 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:17.276675 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:17.277419 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:17.277425 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:17.277431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:17.277592 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:17.278036 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:17.278039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:17.278041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:17.278553 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:17.278705 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:17.279435 (+ 730us) server_negotiation.cc:299] Negotiation successful
0926 14:47:17.279752 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":104,"threads_started":1}
I20260926 14:47:17.280726 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.264295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:45441)
0926 14:47:17.264835 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:17.264879 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:17.265687 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:17.266909 (+ 1222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:17.266917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:17.267337 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:17.267815 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.267841 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.269171 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.269174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.270412 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.270424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.271487 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.271491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.271695 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.272334 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:17.272350 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:17.274113 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:17.276828 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:17.276835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:17.276873 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:17.277266 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:17.277710 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:17.277713 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:17.277715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:17.277906 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:17.278683 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:17.278690 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:17.279071 (+ 381us) client_negotiation.cc:769] Sending connection context
0926 14:47:17.279296 (+ 225us) client_negotiation.cc:240] Negotiation successful
0926 14:47:17.279685 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":155,"threads_started":1}
I20260926 14:47:17.281471 14743 tablet_bootstrap.cc:492] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:47:17.281836 14743 tablet_bootstrap.cc:492] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Bootstrap complete.
I20260926 14:47:17.282991 14743 ts_tablet_manager.cc:1397] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.010s
I20260926 14:47:17.283759 14851 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45781
I20260926 14:47:17.283996 14851 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:17.284569 14851 heartbeater.cc:511] Master 127.0.98.62:45781 requested a full tablet report, sending...
I20260926 14:47:17.285425 14743 raft_consensus.cc:348] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:17.285856 14743 raft_consensus.cc:729] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd97849388ff46e8a6fe07a1c975df91, State: Initialized, Role: FOLLOWER
I20260926 14:47:17.286353 14743 consensus_queue.cc:260] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:17.287271 14743 ts_tablet_manager.cc:1428] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260926 14:47:17.287618 14661 ts_manager.cc:194] Registered new tserver with Master: dd97849388ff46e8a6fe07a1c975df91 (127.0.98.1:41871)
I20260926 14:47:17.294062 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45441
I20260926 14:47:17.297063 14851 heartbeater.cc:503] Master 127.0.98.62:45781 was elected leader, sending a full tablet report...
I20260926 14:47:17.305369 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.291470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41871 (local address 127.0.0.1:33212)
0926 14:47:17.299065 (+ 7595us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:17.299109 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:17.299249 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:17.299957 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:17.299963 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:17.299988 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:17.300131 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.300138 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.301610 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.301613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.303307 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.303322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.304771 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.304774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.304903 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.304946 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 14:47:17.305055 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 14:47:17.305194 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7391,"thread_start_us":120,"threads_started":1}
I20260926 14:47:17.305970 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.292179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33212 (local address 127.0.98.1:41871)
0926 14:47:17.293654 (+ 1475us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:17.293693 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:17.299204 (+ 5511us) server_negotiation.cc:407] Connection header received
0926 14:47:17.299426 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:17.299432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:17.299643 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:17.299789 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:17.300260 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.301426 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.303477 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.304440 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.304551 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.304704 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:17.305724 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0926 14:47:17.305836 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1284,"thread_start_us":103,"threads_started":1}
I20260926 14:47:17.313764 14785 tablet_service.cc:1514] Processing DeleteTablet for tablet 63aeb89161ec4f6fb6af93fc68cb421a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:16 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33212
I20260926 14:47:17.315003 14860 tablet_replica.cc:330] stopping tablet replica
I20260926 14:47:17.315402 14860 raft_consensus.cc:2229] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:47:17.315692 14860 raft_consensus.cc:2258] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:17.316875 14860 ts_tablet_manager.cc:1905] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:17.322101 14860 ts_tablet_manager.cc:1918] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:47:17.322242 14860 log.cc:1196] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal/wals/63aeb89161ec4f6fb6af93fc68cb421a
I20260926 14:47:17.322538 14860 ts_tablet_manager.cc:1939] T 63aeb89161ec4f6fb6af93fc68cb421a P dd97849388ff46e8a6fe07a1c975df91: Deleting consensus metadata
I20260926 14:47:17.324802 14647 catalog_manager.cc:4841] TS dd97849388ff46e8a6fe07a1c975df91 (127.0.98.1:41871): tablet 63aeb89161ec4f6fb6af93fc68cb421a (table test-table [id=870ac701c477473c9254f13f493bdd19]) successfully deleted
W20260926 14:47:17.393540 14855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:17.393828 14855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:17.393900 14855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:17.398180 14855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:17.398253 14855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:17.398339 14855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:47:17.404019 14855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44969
--tserver_master_addrs=127.0.98.62:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:17.405155 14855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:17.406163 14855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:17.415720 14866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:17.415792 14865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:17.415910 14868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:17.416103 14855 server_base.cc:1034] running on GCE node
I20260926 14:47:17.416523 14855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:17.417186 14855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:17.418362 14855 hybrid_clock.cc:648] HybridClock initialized: now 1790434037418328 us; error 45 us; skew 500 ppm
Sep 26 14:47:17 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:17.421229 14855 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:47:17.422129 14855 webserver.cc:460] Webserver started at http://127.0.98.2:44969/ using document root <none> and password file <none>
I20260926 14:47:17.422672 14855 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:17.422750 14855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:17.426553 14855 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:47:17.428655 14875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:17.429914 14855 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:47:17.430023 14855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "747dcdb586f74a30b7ac99f74601971c"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "9ad4b51d360ca1b806bea6b6d3e50ea5"
server_key_iv: "825829a19defeef7de13609a13e66955"
server_key_version: "encryptionkey@0"
I20260926 14:47:17.430562 14855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:17.439654 14855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:17.442914 14855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:17.443095 14855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:17.443676 14855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:17.444485 14855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:17.444532 14855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:17.444574 14855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:17.444602 14855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:17.453189 14855 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40219
I20260926 14:47:17.453222 14988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40219 every 8 connection(s)
I20260926 14:47:17.454211 14855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:47:17.457289 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 14855
I20260926 14:47:17.457744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38117
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45781
--webserver_port=45361
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:47:17 dist-test-slave-s7k6 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:17.469851 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.456043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52109 (local address 127.0.98.62:45781)
0926 14:47:17.456167 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:17.456171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:17.457318 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 14:47:17.458556 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:17.458559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:17.458615 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:17.458694 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:17.460104 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.460647 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.462274 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.462576 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.462674 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.465718 (+ 3044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:17.465722 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:17.465724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:17.465756 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:17.467349 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:17.468067 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:17.468070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:17.468072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:17.468159 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:17.468591 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:17.468594 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:17.468595 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:17.468802 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:17.468899 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:17.469567 (+ 668us) server_negotiation.cc:299] Negotiation successful
0926 14:47:17.469695 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:47:17.471076 14991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.456436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:52109)
0926 14:47:17.457135 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:17.457195 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:17.458278 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:17.458950 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:17.458959 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:17.459430 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:17.459943 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.459957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.460777 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.460781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.462107 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.462120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.462780 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.462786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.463020 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.463642 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:17.463667 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:17.465489 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:17.467518 (+ 2029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:17.467526 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:17.467543 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:17.467945 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:17.468251 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:17.468254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:17.468257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:17.468470 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:17.468915 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:17.468922 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:17.469273 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 14:47:17.469489 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 14:47:17.469864 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":107,"threads_started":1}
I20260926 14:47:17.472337 14989 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45781
I20260926 14:47:17.472610 14989 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:17.473212 14989 heartbeater.cc:511] Master 127.0.98.62:45781 requested a full tablet report, sending...
I20260926 14:47:17.474512 14661 ts_manager.cc:194] Registered new tserver with Master: 747dcdb586f74a30b7ac99f74601971c (127.0.98.2:40219)
I20260926 14:47:17.475175 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52109
W20260926 14:47:17.576027 14992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:17.576295 14992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:17.576340 14992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:17.580539 14992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:17.580605 14992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:17.580667 14992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:47:17.586118 14992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45361
--tserver_master_addrs=127.0.98.62:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:17.587266 14992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:17.588335 14992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:17.598263 14998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:17.598255 15000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:17.598265 14997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:17.598551 14992 server_base.cc:1034] running on GCE node
I20260926 14:47:17.599084 14992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:17.599761 14992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:17.600950 14992 hybrid_clock.cc:648] HybridClock initialized: now 1790434037600931 us; error 37 us; skew 500 ppm
Sep 26 14:47:17 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:17.603920 14992 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:47:17.604969 14992 webserver.cc:460] Webserver started at http://127.0.98.3:45361/ using document root <none> and password file <none>
I20260926 14:47:17.605533 14992 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:17.605614 14992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:17.609607 14992 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:47:17.611901 15007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:17.613221 14992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:47:17.613324 14992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "f1b9cf19ecf0a1883c6437091c651c75"
server_key_iv: "5a64b7e89ec0a139a225bab11a6c7076"
server_key_version: "encryptionkey@0"
I20260926 14:47:17.613865 14992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:17.634296 14992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:17.637921 14992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:17.638082 14992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:17.638689 14992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:17.639581 14992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:17.639636 14992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:17.639678 14992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:17.639691 14992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:17.648717 14992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38117
I20260926 14:47:17.648746 15120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38117 every 8 connection(s)
I20260926 14:47:17.649765 14992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:47:17.655503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 14992
Sep 26 14:47:17 dist-test-slave-s7k6 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:17.663738 14853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.651662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47347 (local address 127.0.98.62:45781)
0926 14:47:17.651793 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:17.651796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:17.652714 (+ 918us) server_negotiation.cc:407] Connection header received
0926 14:47:17.653982 (+ 1268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:17.653986 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:17.654046 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:17.654118 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:17.655392 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.655963 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.657387 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.657608 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.657698 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.660221 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:17.660224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:17.660226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:17.660249 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:17.661600 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:17.662214 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:17.662217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:17.662218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:17.662288 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:17.662624 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:17.662627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:17.662628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:17.662847 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:17.662967 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:17.663450 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 14:47:17.663579 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:47:17.664011 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.656150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42418 (local address 127.0.98.62:45781)
0926 14:47:17.656377 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:17.656381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:17.656479 (+ 98us) server_negotiation.cc:407] Connection header received
0926 14:47:17.656587 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:17.656590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:17.656640 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:17.656725 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:17.657455 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.658039 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.659048 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.659281 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.659369 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.660678 (+ 1309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:17.660681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:17.660683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:17.660716 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:17.662214 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:17.662593 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:17.662596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:17.662598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:17.662677 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:17.662941 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:17.662944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:17.662946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:17.663101 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:17.663295 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:17.663473 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 14:47:17.663814 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":158,"spinlock_wait_cycles":1920,"thread_start_us":66,"threads_started":1}
I20260926 14:47:17.664695 15123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.652058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:47347)
0926 14:47:17.652561 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:17.652607 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:17.653768 (+ 1161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:17.654301 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:17.654309 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:17.654693 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:17.655227 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.655242 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.656089 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.656092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.657265 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.657276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.657764 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.657768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.657939 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.658506 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:17.658523 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:17.660050 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:17.661720 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:17.661726 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:17.661744 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:17.662122 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:17.662386 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:17.662389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:17.662391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:17.662533 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:17.662975 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:17.662982 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:17.663230 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 14:47:17.663407 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 14:47:17.663738 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":113,"threads_started":1}
I20260926 14:47:17.665844 15121 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45781
I20260926 14:47:17.666081 15121 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:17.666567 15121 heartbeater.cc:511] Master 127.0.98.62:45781 requested a full tablet report, sending...
I20260926 14:47:17.667615 14661 ts_manager.cc:194] Registered new tserver with Master: 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117)
I20260926 14:47:17.668143 14661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47347
I20260926 14:47:17.677636 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:47:17.682003 15125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.678271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.0.98.62:45781)
0926 14:47:17.678397 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:17.678401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:17.678579 (+ 178us) server_negotiation.cc:407] Connection header received
0926 14:47:17.678711 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:17.678713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:17.678922 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:17.679015 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:47:17.679244 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.679761 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.680655 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.680897 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.680970 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.681262 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:47:17.681676 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:47:17.681764 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:17.681810 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 14:47:17.681890 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:47:17.682649 14661 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:47:17.686000 14661 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:47:17.694432 14785 tablet_service.cc:1467] Processing CreateTablet for tablet 8cca4b0cda214a9c9451e74f1d7aa348 (DEFAULT_TABLE table=test-table [id=b96d9b90891a4bd8ab4454afa8d56ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:17.695197 14785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cca4b0cda214a9c9451e74f1d7aa348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:17.698050 14743 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Bootstrap starting.
I20260926 14:47:17.698206 14858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.693152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38117 (local address 127.0.0.1:38062)
0926 14:47:17.693483 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:17.693500 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:17.693581 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:17.694513 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:17.694516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:17.694533 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:17.694639 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.694646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.696084 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.696088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.696935 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.696944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.697832 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.697836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.697919 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.697948 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 14:47:17.697995 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 14:47:17.698073 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":1}
I20260926 14:47:17.699064 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.693326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.0.98.3:38117)
0926 14:47:17.693993 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:17.694010 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:17.694035 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:47:17.694116 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:17.694122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:17.694364 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:17.694528 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:17.694849 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.695944 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.697050 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.697689 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.698584 (+ 895us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.698743 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:17.698824 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 14:47:17.698924 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":215,"threads_started":1}
I20260926 14:47:17.699131 14743 tablet_bootstrap.cc:654] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:17.699590 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.693439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40219 (local address 127.0.0.1:34732)
0926 14:47:17.693662 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:17.693676 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:17.693764 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:17.694801 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:17.694803 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:17.694821 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:17.694938 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.694944 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.696424 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.696427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.697479 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.697490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.699164 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.699168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.699270 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.699304 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 14:47:17.699385 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 14:47:17.699455 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":100,"threads_started":1}
I20260926 14:47:17.699657 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.693852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34732 (local address 127.0.98.2:40219)
0926 14:47:17.694143 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:17.694154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:17.694177 (+ 23us) server_negotiation.cc:407] Connection header received
0926 14:47:17.694240 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:17.694244 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:17.694389 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:17.694530 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:17.695050 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.696273 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.697612 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.699038 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.699152 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.699291 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:17.699426 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 14:47:17.699525 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":119,"threads_started":1}
I20260926 14:47:17.700364 14743 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: No bootstrap required, opened a new log
I20260926 14:47:17.700449 14743 ts_tablet_manager.cc:1397] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 14:47:17.701112 14743 raft_consensus.cc:348] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:17.701212 14743 raft_consensus.cc:374] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:17.701238 14743 raft_consensus.cc:729] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd97849388ff46e8a6fe07a1c975df91, State: Initialized, Role: FOLLOWER
I20260926 14:47:17.701359 14743 consensus_queue.cc:260] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:17.701597 14743 ts_tablet_manager.cc:1428] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:47:17.702646 15055 tablet_service.cc:1467] Processing CreateTablet for tablet 8cca4b0cda214a9c9451e74f1d7aa348 (DEFAULT_TABLE table=test-table [id=b96d9b90891a4bd8ab4454afa8d56ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:17.703709 15055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cca4b0cda214a9c9451e74f1d7aa348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:17.703665 14923 tablet_service.cc:1467] Processing CreateTablet for tablet 8cca4b0cda214a9c9451e74f1d7aa348 (DEFAULT_TABLE table=test-table [id=b96d9b90891a4bd8ab4454afa8d56ff8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:17.704722 14923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cca4b0cda214a9c9451e74f1d7aa348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:17.710863 15134 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Bootstrap starting.
I20260926 14:47:17.711521 15135 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: Bootstrap starting.
I20260926 14:47:17.713043 15134 tablet_bootstrap.cc:654] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:17.713639 15135 tablet_bootstrap.cc:654] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:17.714087 15134 log.cc:824] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:17.714359 15135 log.cc:824] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:17.716284 15135 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: No bootstrap required, opened a new log
I20260926 14:47:17.716284 15134 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: No bootstrap required, opened a new log
I20260926 14:47:17.716493 15134 ts_tablet_manager.cc:1397] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:47:17.716493 15135 ts_tablet_manager.cc:1397] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:47:17.720533 15134 raft_consensus.cc:348] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:17.720533 15135 raft_consensus.cc:348] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:17.720847 15134 raft_consensus.cc:374] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:17.720907 15134 raft_consensus.cc:729] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b59a31f1cef4b1691b5f40f3fa2d9eb, State: Initialized, Role: FOLLOWER
I20260926 14:47:17.720846 15135 raft_consensus.cc:374] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:17.720969 15135 raft_consensus.cc:729] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747dcdb586f74a30b7ac99f74601971c, State: Initialized, Role: FOLLOWER
I20260926 14:47:17.721406 15134 consensus_queue.cc:260] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:17.721405 15135 consensus_queue.cc:260] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:17.722115 14989 heartbeater.cc:503] Master 127.0.98.62:45781 was elected leader, sending a full tablet report...
I20260926 14:47:17.722348 15121 heartbeater.cc:503] Master 127.0.98.62:45781 was elected leader, sending a full tablet report...
I20260926 14:47:17.722497 15134 ts_tablet_manager.cc:1428] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260926 14:47:17.722497 15135 ts_tablet_manager.cc:1428] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 14:47:17.756990 14857 raft_consensus.cc:482] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:47:17.757215 14857 raft_consensus.cc:504] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:17.758402 14857 leader_election.cc:288] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117), 747dcdb586f74a30b7ac99f74601971c (127.0.98.2:40219)
I20260926 14:47:17.762354 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.758714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52141 (local address 127.0.98.3:38117)
0926 14:47:17.758845 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:17.758849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:17.758963 (+ 114us) server_negotiation.cc:407] Connection header received
0926 14:47:17.759057 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:17.759060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:17.759120 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:17.759194 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:17.759482 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.760045 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.761336 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.761755 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.761848 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.761921 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:17.762145 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 14:47:17.762215 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:47:17.762358 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.758826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54685 (local address 127.0.98.2:40219)
0926 14:47:17.758960 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:17.758965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:17.759107 (+ 142us) server_negotiation.cc:407] Connection header received
0926 14:47:17.759217 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:17.759220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:17.759270 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:17.759354 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:17.759611 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.760108 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.761338 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.761754 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.761848 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.761914 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:17.762150 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 14:47:17.762226 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 14:47:17.762364 14854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.758622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38117 (local address 127.0.98.1:52141)
0926 14:47:17.758829 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:17.758844 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:17.758958 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:17.759213 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:17.759216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:17.759234 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:17.759367 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.759374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.760164 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.760168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.761202 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.761212 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.761884 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.761888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.762020 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.762049 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 14:47:17.762118 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:47:17.762242 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":164}
I20260926 14:47:17.762403 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.758670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40219 (local address 127.0.98.1:54685)
0926 14:47:17.759010 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:17.759024 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:17.759131 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:17.759369 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:17.759371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:17.759391 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:17.759507 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.759514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.760227 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.760231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.761203 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:17.761213 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.761891 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.761895 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:17.762025 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.762050 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:47:17.762116 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:47:17.762218 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":58,"thread_start_us":112,"threads_started":1}
I20260926 14:47:17.763073 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cca4b0cda214a9c9451e74f1d7aa348" candidate_uuid: "dd97849388ff46e8a6fe07a1c975df91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" is_pre_election: true
I20260926 14:47:17.763185 14943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cca4b0cda214a9c9451e74f1d7aa348" candidate_uuid: "dd97849388ff46e8a6fe07a1c975df91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747dcdb586f74a30b7ac99f74601971c" is_pre_election: true
I20260926 14:47:17.763409 15075 raft_consensus.cc:2454] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd97849388ff46e8a6fe07a1c975df91 in term 0.
I20260926 14:47:17.763447 14943 raft_consensus.cc:2454] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd97849388ff46e8a6fe07a1c975df91 in term 0.
W20260926 14:47:17.763460 14852 tablet.cc:2369] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:47:17.763911 14740 leader_election.cc:302] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747dcdb586f74a30b7ac99f74601971c, dd97849388ff46e8a6fe07a1c975df91; no voters:
I20260926 14:47:17.764190 14857 raft_consensus.cc:2781] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:47:17.764281 14857 raft_consensus.cc:482] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:47:17.764324 14857 raft_consensus.cc:3037] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:17.765376 14857 raft_consensus.cc:504] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:17.765697 14857 leader_election.cc:288] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 1 election: Requested vote from peers 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117), 747dcdb586f74a30b7ac99f74601971c (127.0.98.2:40219)
I20260926 14:47:17.766057 15075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cca4b0cda214a9c9451e74f1d7aa348" candidate_uuid: "dd97849388ff46e8a6fe07a1c975df91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb"
I20260926 14:47:17.766077 14943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cca4b0cda214a9c9451e74f1d7aa348" candidate_uuid: "dd97849388ff46e8a6fe07a1c975df91" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "747dcdb586f74a30b7ac99f74601971c"
I20260926 14:47:17.766182 15075 raft_consensus.cc:3037] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:17.766196 14943 raft_consensus.cc:3037] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:17.767302 15075 raft_consensus.cc:2454] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd97849388ff46e8a6fe07a1c975df91 in term 1.
I20260926 14:47:17.767318 14943 raft_consensus.cc:2454] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd97849388ff46e8a6fe07a1c975df91 in term 1.
I20260926 14:47:17.767589 14740 leader_election.cc:302] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747dcdb586f74a30b7ac99f74601971c, dd97849388ff46e8a6fe07a1c975df91; no voters:
I20260926 14:47:17.767832 14857 raft_consensus.cc:2781] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:47:17.768035 14857 raft_consensus.cc:686] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 1 LEADER]: Becoming Leader. State: Replica: dd97849388ff46e8a6fe07a1c975df91, State: Running, Role: LEADER
I20260926 14:47:17.768273 14857 consensus_queue.cc:237] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:17.771245 14661 catalog_manager.cc:5495] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd97849388ff46e8a6fe07a1c975df91 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dd97849388ff46e8a6fe07a1c975df91" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } health_report { overall_health: HEALTHY } } }
I20260926 14:47:17.799358 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.795481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33218 (local address 127.0.98.1:41871)
0926 14:47:17.795618 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:17.795622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:17.795636 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:47:17.795800 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:17.795803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:17.795862 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:17.795986 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:47:17.796265 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.797062 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.797920 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:17.798146 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:17.798227 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:17.798449 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:47:17.798992 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:47:17.799086 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:17.799152 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 14:47:17.799208 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:47:17.803411 14943 raft_consensus.cc:1261] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Refusing update from remote peer dd97849388ff46e8a6fe07a1c975df91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:17.803411 15075 raft_consensus.cc:1261] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 1 FOLLOWER]: Refusing update from remote peer dd97849388ff46e8a6fe07a1c975df91: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:17.804126 15141 consensus_queue.cc:1035] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:47:17.804294 14857 consensus_queue.cc:1035] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:47:17.811646 15144 mvcc.cc:205] Tried to move back new op lower bound from 7333617818836258816 to 7333617818701324288. Current Snapshot: MvccSnapshot[applied={T|T < 7333617818836258816}]
I20260926 14:47:17.812239 15143 mvcc.cc:205] Tried to move back new op lower bound from 7333617818836258816 to 7333617818701324288. Current Snapshot: MvccSnapshot[applied={T|T < 7333617818836258816 or (T in {7333617818836258816})}]
I20260926 14:47:17.814136 15145 mvcc.cc:205] Tried to move back new op lower bound from 7333617818836258816 to 7333617818701324288. Current Snapshot: MvccSnapshot[applied={T|T < 7333617818836258816 or (T in {7333617818836258816})}]
I20260926 14:47:17.819967 14661 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:47:17.820142 14661 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:47:17.822584 14661 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Sending DeleteTablet for 3 replicas of tablet 8cca4b0cda214a9c9451e74f1d7aa348
I20260926 14:47:17.823417 15055 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cca4b0cda214a9c9451e74f1d7aa348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38062
I20260926 14:47:17.823405 14923 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cca4b0cda214a9c9451e74f1d7aa348 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:17 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34732
I20260926 14:47:17.823997 15152 tablet_replica.cc:330] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: stopping tablet replica
I20260926 14:47:17.824162 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 14701
I20260926 14:47:17.824831 15152 raft_consensus.cc:2229] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:47:17.825145 15152 raft_consensus.cc:2258] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:17.826426 15152 ts_tablet_manager.cc:1905] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:17.828619 15152 ts_tablet_manager.cc:1918] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:47:17.828725 15152 log.cc:1196] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/8cca4b0cda214a9c9451e74f1d7aa348
I20260926 14:47:17.828778 15153 tablet_replica.cc:330] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: stopping tablet replica
I20260926 14:47:17.829057 15152 ts_tablet_manager.cc:1939] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Deleting consensus metadata
I20260926 14:47:17.829137 15153 raft_consensus.cc:2229] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:47:17.829555 15153 raft_consensus.cc:2258] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:17.830459 14649 catalog_manager.cc:4841] TS 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117): tablet 8cca4b0cda214a9c9451e74f1d7aa348 (table test-table [id=b96d9b90891a4bd8ab4454afa8d56ff8]) successfully deleted
W20260926 14:47:17.830505 14647 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41871 (error 108)
I20260926 14:47:17.830783 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 14855
I20260926 14:47:17.832911 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.832602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41871 (local address 127.0.0.1:33224)
0926 14:47:17.832762 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:17.832828 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 14:47:17.833070 14647 catalog_manager.cc:4584] TS dd97849388ff46e8a6fe07a1c975df91 (127.0.98.1:41871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8cca4b0cda214a9c9451e74f1d7aa348: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41871: connect: Connection refused (error 111)
I20260926 14:47:17.836815 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 14992
I20260926 14:47:17.837584 15130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:17.836955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40219 (local address 127.0.0.1:34748)
0926 14:47:17.837443 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:17.837499 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412}
I20260926 14:47:17.842025 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 14630
I20260926 14:47:17.847676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45781
--webserver_interface=127.0.98.62
--webserver_port=38413
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45781
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:17.960682 15155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:17.960983 15155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:17.961027 15155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:17.965206 15155 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:47:17.965272 15155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:17.965299 15155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:17.965314 15155 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:47:17.965346 15155 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:47:17.970894 15155 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:17.972029 15155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:17.973105 15155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:47:17.981906 15155 server_base.cc:1034] running on GCE node
W20260926 14:47:17.981878 15160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:17.981833 15163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:17.981833 15161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:17.982645 15155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:17.983987 15155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:17.985163 15155 hybrid_clock.cc:648] HybridClock initialized: now 1790434037985144 us; error 35 us; skew 500 ppm
Sep 26 14:47:17 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434037, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:17.988001 15155 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:47:17.988925 15155 webserver.cc:460] Webserver started at http://127.0.98.62:38413/ using document root <none> and password file <none>
I20260926 14:47:17.989475 15155 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:17.989547 15155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:17.993559 15155 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:47:17.995810 15170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:17.997175 15155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:47:17.997318 15155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "71fbb757681f441fb1aeab7bbebddbc0"
format_stamp: "Formatted at 2026-09-26 14:47:15 on dist-test-slave-s7k6"
server_key: "d50dd0c3b4820eb1c40350f334221b43"
server_key_iv: "b95859022b0524837e8179ddc4592b7e"
server_key_version: "encryptionkey@0"
I20260926 14:47:17.997895 15155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:18.018229 15155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:18.021353 15155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:18.021544 15155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:18.028399 15155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45781
I20260926 14:47:18.028651 15222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45781 every 8 connection(s)
I20260926 14:47:18.029488 15155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:47:18.033685 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 15155
I20260926 14:47:18.034180 15223 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:47:18.034173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:41871
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45781
--webserver_port=37045
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:18.036379 15223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Bootstrap starting.
I20260926 14:47:18.053813 15223 log.cc:824] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:18.065486 15223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:47:18.065869 15223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0: Bootstrap complete.
I20260926 14:47:18.069867 15223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:18.070281 15223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71fbb757681f441fb1aeab7bbebddbc0, State: Initialized, Role: FOLLOWER
I20260926 14:47:18.070786 15223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:18.070914 15223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:47:18.070982 15223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:47:18.071075 15223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 2 FOLLOWER]: Advancing to term 3
I20260926 14:47:18.073019 15223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:18.073369 15223 leader_election.cc:302] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71fbb757681f441fb1aeab7bbebddbc0; no voters:
I20260926 14:47:18.073704 15223 leader_election.cc:288] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 14:47:18.073817 15227 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 14:47:18.074013 15227 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [term 3 LEADER]: Becoming Leader. State: Replica: 71fbb757681f441fb1aeab7bbebddbc0, State: Running, Role: LEADER
I20260926 14:47:18.074338 15227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } }
I20260926 14:47:18.075099 15223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:47:18.075454 15227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71fbb757681f441fb1aeab7bbebddbc0. Latest consensus state: current_term: 3 leader_uuid: "71fbb757681f441fb1aeab7bbebddbc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } } }
I20260926 14:47:18.075585 15227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:18.077096 15231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:47:18.076309 15228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "71fbb757681f441fb1aeab7bbebddbc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71fbb757681f441fb1aeab7bbebddbc0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45781 } } }
I20260926 14:47:18.077689 15228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71fbb757681f441fb1aeab7bbebddbc0 [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:18.083096 15231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:47:18.083774 15231 catalog_manager.cc:1259] Loaded cluster ID: 0ca8a5c109c849bbafed755ef2b3149e
I20260926 14:47:18.083839 15231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 14:47:18.084830 15231 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6a1800c9ad google::LogMessage::Fail() at ??:0
@ 0x7f6a1800f88a google::LogMessage::SendToLog() at ??:0
@ 0x7f6a1800c4de google::LogMessage::Flush() at ??:0
@ 0x7f6a1800da19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6a2294bfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f6a2294cd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6a22949579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f6a2298d640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6a1a25e3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6a1a25ec83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f6a1a2603f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f6a1a24f512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6a1b8936db start_thread at ??:0
@ 0x7f6a16b0a61f clone at ??:0
W20260926 14:47:18.176105 15225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:18.176411 15225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:18.176594 15225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:18.181305 15225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:18.181380 15225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:18.181445 15225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:47:18.187124 15225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:41871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37045
--tserver_master_addrs=127.0.98.62:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:18.188215 15225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:18.189272 15225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:18.199373 15265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:18.199373 15264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:18.199378 15267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:18.199994 15225 server_base.cc:1034] running on GCE node
I20260926 14:47:18.200637 15225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:18.201467 15225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:18.202683 15225 hybrid_clock.cc:648] HybridClock initialized: now 1790434038202663 us; error 33 us; skew 500 ppm
Sep 26 14:47:18 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:18.206563 15225 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:47:18.207837 15225 webserver.cc:460] Webserver started at http://127.0.98.1:37045/ using document root <none> and password file <none>
I20260926 14:47:18.208611 15225 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:18.208714 15225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:18.214123 15225 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.002s
I20260926 14:47:18.216991 15274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:18.218674 15225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:47:18.218853 15225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "dd97849388ff46e8a6fe07a1c975df91"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "390988d1e209ae5d9f5319854e7e52aa"
server_key_iv: "6b98c6a7312340f006a9e2599ebbc56d"
server_key_version: "encryptionkey@0"
I20260926 14:47:18.219597 15225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:18.243438 15225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:18.247527 15225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:18.247764 15225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:18.248575 15225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:18.249840 15281 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:47:18.254467 15225 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:47:18.254557 15225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 14:47:18.254627 15225 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:47:18.257085 15225 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:47:18.257159 15225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260926 14:47:18.257460 15281 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Bootstrap starting.
I20260926 14:47:18.270298 15225 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41871
I20260926 14:47:18.271660 15225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:47:18.272887 15388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41871 every 8 connection(s)
I20260926 14:47:18.275053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 15225
I20260926 14:47:18.275522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40219
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45781
--webserver_port=44969
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:18.282088 15281 log.cc:824] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:18.293733 15281 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:47:18.294220 15281 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Bootstrap complete.
I20260926 14:47:18.295652 15281 ts_tablet_manager.cc:1397] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Time spent bootstrapping tablet: real 0.038s user 0.029s sys 0.004s
I20260926 14:47:18.299062 15281 raft_consensus.cc:348] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:18.299659 15281 raft_consensus.cc:729] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd97849388ff46e8a6fe07a1c975df91, State: Initialized, Role: FOLLOWER
I20260926 14:47:18.300205 15281 consensus_queue.cc:260] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:18.301302 15281 ts_tablet_manager.cc:1428] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260926 14:47:18.327184 15391 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:47:18.275107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:42843)
0926 14:47:18.278924 (+ 3817us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:18.279008 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:18.280296 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:18.326330 (+ 46034us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3476,"thread_start_us":163,"threads_started":1}
W20260926 14:47:18.327721 15389 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45781 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45781: Client connection negotiation failed: client connection to 127.0.98.62:45781: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:47:18.328523 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:18.328263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:54325)
0926 14:47:18.328387 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:18.328460 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 14:47:18.329144 15391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:18.328945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:53999)
0926 14:47:18.329046 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:18.329091 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:47:18.329296 15389 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:47:18.402448 15392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:18.402750 15392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:18.402840 15392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:18.407186 15392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:18.407256 15392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:18.407342 15392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:47:18.412966 15392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40219
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44969
--tserver_master_addrs=127.0.98.62:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15392
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:18.414062 15392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:18.415136 15392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:18.424754 15400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:18.424927 15402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:18.424927 15399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:18.425226 15392 server_base.cc:1034] running on GCE node
I20260926 14:47:18.425707 15392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:18.426396 15392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:18.427577 15392 hybrid_clock.cc:648] HybridClock initialized: now 1790434038427554 us; error 33 us; skew 500 ppm
Sep 26 14:47:18 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:18.430362 15392 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:47:18.431356 15392 webserver.cc:460] Webserver started at http://127.0.98.2:44969/ using document root <none> and password file <none>
I20260926 14:47:18.431921 15392 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:18.431984 15392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:18.435964 15392 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 14:47:18.438246 15409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:18.439611 15392 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:47:18.439733 15392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "747dcdb586f74a30b7ac99f74601971c"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "9ad4b51d360ca1b806bea6b6d3e50ea5"
server_key_iv: "825829a19defeef7de13609a13e66955"
server_key_version: "encryptionkey@0"
I20260926 14:47:18.440311 15392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:18.458628 15392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:18.461797 15392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:18.461987 15392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:18.462605 15392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:18.463757 15416 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:47:18.468087 15392 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:47:18.468165 15392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 14:47:18.468233 15392 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:47:18.470099 15392 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:47:18.470153 15392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 14:47:18.470482 15416 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: Bootstrap starting.
I20260926 14:47:18.477031 15416 log.cc:824] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:18.479784 15392 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40219
I20260926 14:47:18.479835 15524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40219 every 8 connection(s)
I20260926 14:47:18.480691 15392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:47:18.484337 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:18.482735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:59303)
0926 14:47:18.483306 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:18.483459 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":140,"threads_started":1}
W20260926 14:47:18.484814 15525 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45781 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45781: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
I20260926 14:47:18.485555 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:18.485297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:55991)
0926 14:47:18.485426 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:18.485487 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 14:47:18.485844 15416 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:47:18.486276 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:18.486059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:59549)
0926 14:47:18.486165 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:18.486215 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:47:18.486276 15416 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: Bootstrap complete.
W20260926 14:47:18.486518 15525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:47:18.486692 15416 ts_tablet_manager.cc:1397] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: Time spent bootstrapping tablet: real 0.016s user 0.016s sys 0.000s
I20260926 14:47:18.487586 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 15392
I20260926 14:47:18.487999 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38117
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45781
--webserver_port=45361
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:18.490151 15416 raft_consensus.cc:348] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:18.490587 15416 raft_consensus.cc:729] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747dcdb586f74a30b7ac99f74601971c, State: Initialized, Role: FOLLOWER
I20260926 14:47:18.491060 15416 consensus_queue.cc:260] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:18.492058 15416 ts_tablet_manager.cc:1428] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:47:18.492250 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:18.491920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:44277)
0926 14:47:18.492109 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:18.492171 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 14:47:18.492987 15527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:18.492725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:39795)
0926 14:47:18.492834 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:18.492909 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:47:18.602172 15528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:18.602443 15528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:18.602494 15528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:18.606648 15528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:18.606734 15528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:18.606837 15528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:47:18.612260 15528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38117
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45361
--tserver_master_addrs=127.0.98.62:45781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:18.613341 15528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:18.614447 15528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:18.623960 15535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:18.623961 15534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:18.623958 15537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:18.624465 15528 server_base.cc:1034] running on GCE node
I20260926 14:47:18.624892 15528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:18.625553 15528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:18.626709 15528 hybrid_clock.cc:648] HybridClock initialized: now 1790434038626686 us; error 37 us; skew 500 ppm
Sep 26 14:47:18 dist-test-slave-s7k6 krb5kdc[14081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:18.629438 15528 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:47:18.630357 15528 webserver.cc:460] Webserver started at http://127.0.98.3:45361/ using document root <none> and password file <none>
I20260926 14:47:18.630973 15528 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:18.631054 15528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:18.635085 15528 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:47:18.637404 15544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:18.638989 15528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:47:18.639114 15528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb"
format_stamp: "Formatted at 2026-09-26 14:47:16 on dist-test-slave-s7k6"
server_key: "f1b9cf19ecf0a1883c6437091c651c75"
server_key_iv: "5a64b7e89ec0a139a225bab11a6c7076"
server_key_version: "encryptionkey@0"
I20260926 14:47:18.639704 15528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:18.654570 15528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:18.657701 15528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:18.657914 15528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:18.658535 15528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:18.659475 15528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:18.659528 15528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:18.659591 15528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:18.659619 15528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:18.668479 15528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38117
I20260926 14:47:18.668560 15657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38117 every 8 connection(s)
I20260926 14:47:18.669524 15528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:47:18.673310 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:18.671787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:48501)
0926 14:47:18.672462 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:18.672585 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":106,"threads_started":1}
W20260926 14:47:18.673817 15658 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45781 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:45781: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
I20260926 14:47:18.674424 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:18.674201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:45465)
0926 14:47:18.674312 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:18.674361 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 14:47:18.675170 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:18.674915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:52895)
0926 14:47:18.675044 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:18.675101 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 14:47:18.675298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 15528
W20260926 14:47:18.675688 15658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:47:19.330415 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:19.329962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:58383)
0926 14:47:19.330221 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:19.330291 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":79,"threads_started":1}
I20260926 14:47:19.494139 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:19.493707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:50901)
0926 14:47:19.493993 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:19.494053 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
I20260926 14:47:19.676791 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:19.676334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:40935)
0926 14:47:19.676591 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:19.676676 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":73,"threads_started":1}
I20260926 14:47:19.698026 15665 raft_consensus.cc:482] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:47:19.698259 15665 raft_consensus.cc:504] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:19.699422 15665 leader_election.cc:288] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117), 747dcdb586f74a30b7ac99f74601971c (127.0.98.2:40219)
Sep 26 14:47:19 dist-test-slave-s7k6 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 14:47:19 dist-test-slave-s7k6 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 14:47:19.713413 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:19.699692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38117 (local address 127.0.98.1:34167)
0926 14:47:19.700135 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:19.700147 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:19.700235 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:19.701553 (+ 1318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:19.701557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:19.701826 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:19.702339 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:19.702354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:19.703777 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:19.703781 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:19.704685 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:19.704694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:19.705402 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:19.705406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:19.705656 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:19.706318 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:19.706330 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:19.708352 (+ 2022us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:19.710557 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:19.710560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:19.710562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:19.710778 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:19.711188 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:19.711192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:19.711194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:19.711434 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:19.712261 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:19.712268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:19.712640 (+ 372us) client_negotiation.cc:769] Sending connection context
0926 14:47:19.712857 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 14:47:19.713272 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":370,"spinlock_wait_cycles":9728,"thread_start_us":128,"threads_started":1}
I20260926 14:47:19.713413 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:19.699694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40219 (local address 127.0.98.1:50735)
0926 14:47:19.699792 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:19.699807 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:19.699914 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:19.701286 (+ 1372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:19.701292 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:19.701857 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:19.702335 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:19.702351 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:19.703903 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:19.703907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:19.704684 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:19.704694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:19.705396 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:19.705400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:19.705656 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:19.706278 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:19.706295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:19.708120 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:19.710300 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:19.710306 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:19.710320 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:19.710584 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:19.711188 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:19.711192 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:19.711194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:19.711434 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:19.712261 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:19.712268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:19.712640 (+ 372us) client_negotiation.cc:769] Sending connection context
0926 14:47:19.712857 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 14:47:19.713209 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":173,"spinlock_wait_cycles":44416}
I20260926 14:47:19.713709 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:19.699865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34167 (local address 127.0.98.3:38117)
0926 14:47:19.700262 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:19.700285 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:19.700324 (+ 39us) server_negotiation.cc:407] Connection header received
0926 14:47:19.700992 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:19.700997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:19.701324 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:19.701592 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:19.702544 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:19.703629 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:19.704810 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:19.705284 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:19.705435 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:19.708533 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:19.708539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:19.708551 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:19.708580 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:19.710450 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:19.710876 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:19.710882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:19.710887 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:19.711031 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:19.711561 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:19.711565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:19.711567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:19.712125 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:19.712273 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:19.713008 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 14:47:19.713318 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":70,"threads_started":1}
I20260926 14:47:19.714025 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:19.699869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50735 (local address 127.0.98.2:40219)
0926 14:47:19.700278 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:19.700293 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:19.700333 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:47:19.700564 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:19.700569 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:19.700910 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:19.701217 (+ 307us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:19.702531 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:19.703757 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:19.704804 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:19.705276 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:19.705434 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:19.708392 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:19.708399 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:19.708418 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:19.708453 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:19.710184 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:19.710711 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:19.710716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:19.710721 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:19.710975 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:19.711562 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:19.711565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:19.711567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:19.712167 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:19.712274 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:19.713008 (+ 734us) server_negotiation.cc:299] Negotiation successful
0926 14:47:19.713318 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":73,"threads_started":1}
I20260926 14:47:19.715221 15478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cca4b0cda214a9c9451e74f1d7aa348" candidate_uuid: "dd97849388ff46e8a6fe07a1c975df91" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "747dcdb586f74a30b7ac99f74601971c" is_pre_election: true
I20260926 14:47:19.715575 15478 raft_consensus.cc:2454] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd97849388ff46e8a6fe07a1c975df91 in term 1.
I20260926 14:47:19.716598 15278 leader_election.cc:302] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 747dcdb586f74a30b7ac99f74601971c, dd97849388ff46e8a6fe07a1c975df91; no voters:
W20260926 14:47:19.716789 15277 leader_election.cc:341] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117): Not found: Tablet not found: 8cca4b0cda214a9c9451e74f1d7aa348
I20260926 14:47:19.716918 15665 raft_consensus.cc:2781] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 14:47:19.717015 15665 raft_consensus.cc:482] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:47:19.717072 15665 raft_consensus.cc:3037] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:47:19.719537 15665 raft_consensus.cc:504] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:19.719913 15665 leader_election.cc:288] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 2 election: Requested vote from peers 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117), 747dcdb586f74a30b7ac99f74601971c (127.0.98.2:40219)
I20260926 14:47:19.720323 15478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cca4b0cda214a9c9451e74f1d7aa348" candidate_uuid: "dd97849388ff46e8a6fe07a1c975df91" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "747dcdb586f74a30b7ac99f74601971c"
I20260926 14:47:19.720458 15478 raft_consensus.cc:3037] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 1 FOLLOWER]: Advancing to term 2
W20260926 14:47:19.720599 15277 leader_election.cc:341] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117): Not found: Tablet not found: 8cca4b0cda214a9c9451e74f1d7aa348
I20260926 14:47:19.722775 15478 raft_consensus.cc:2454] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd97849388ff46e8a6fe07a1c975df91 in term 2.
I20260926 14:47:19.723075 15278 leader_election.cc:302] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 747dcdb586f74a30b7ac99f74601971c, dd97849388ff46e8a6fe07a1c975df91; no voters: 2b59a31f1cef4b1691b5f40f3fa2d9eb
I20260926 14:47:19.723264 15665 raft_consensus.cc:2781] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:47:19.723460 15665 raft_consensus.cc:686] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [term 2 LEADER]: Becoming Leader. State: Replica: dd97849388ff46e8a6fe07a1c975df91, State: Running, Role: LEADER
I20260926 14:47:19.723802 15665 consensus_queue.cc:237] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:19.724144 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:19.723873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:42273)
0926 14:47:19.723990 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:19.724043 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 14:47:20.194509 15478 raft_consensus.cc:1261] T 8cca4b0cda214a9c9451e74f1d7aa348 P 747dcdb586f74a30b7ac99f74601971c [term 2 FOLLOWER]: Refusing update from remote peer dd97849388ff46e8a6fe07a1c975df91: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260926 14:47:20.195346 15665 consensus_queue.cc:1035] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 [LEADER]: Connected to new peer: Peer: permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260926 14:47:20.197728 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:20.197245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:40063)
0926 14:47:20.197529 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:20.197609 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":60,"threads_started":1}
I20260926 14:47:20.200250 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:20.199972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:46665)
0926 14:47:20.200099 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:20.200166 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 14:47:20.200670 15277 consensus_peers.cc:487] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91 -> Peer 2b59a31f1cef4b1691b5f40f3fa2d9eb (127.0.98.3:38117): Couldn't send request to peer 2b59a31f1cef4b1691b5f40f3fa2d9eb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cca4b0cda214a9c9451e74f1d7aa348. This is attempt 1: this message will repeat every 5th retry.
I20260926 14:47:20.200905 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:20.200698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:51105)
0926 14:47:20.200804 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:20.200849 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 14:47:20.678042 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:20.677592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:43067)
0926 14:47:20.677857 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:20.677924 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260926 14:47:20.743202 15679 ts_tablet_manager.cc:927] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Initiating tablet copy from peer dd97849388ff46e8a6fe07a1c975df91 (127.0.98.1:41871)
I20260926 14:47:20.743856 15679 tablet_copy_client.cc:316] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.1:41871
Sep 26 14:47:20 dist-test-slave-s7k6 krb5kdc[14081](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 14:47:20.753749 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:20.744397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41871 (local address 127.0.98.3:53421)
0926 14:47:20.744514 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:20.744536 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:20.744647 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:20.745291 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:20.745296 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:20.745671 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:20.745793 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:20.745801 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:20.746749 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:20.746769 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:20.747412 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:20.747421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:20.747992 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:20.747995 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:20.748087 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:20.748592 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:20.748603 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:20.750079 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:20.752028 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:20.752037 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:20.752043 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:20.752289 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:20.752608 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:20.752610 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:20.752611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:20.752695 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:20.753199 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:20.753206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:20.753322 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 14:47:20.753445 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 14:47:20.753589 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 14:47:20.753787 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:20.744517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53421 (local address 127.0.98.1:41871)
0926 14:47:20.744891 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:20.744898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:20.744919 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:47:20.744962 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:20.744966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:20.745146 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:20.745276 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:20.745895 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:20.746640 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:20.747508 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:20.747867 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:20.747947 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:20.750236 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:20.750250 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:20.750267 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:20.750296 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:20.751900 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:20.752420 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:20.752426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:20.752427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:20.752525 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:20.752797 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:20.752800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:20.752801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:20.753077 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:20.753212 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:20.753521 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 14:47:20.753654 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":52,"threads_started":1}
I20260926 14:47:20.754915 15363 tablet_copy_service.cc:140] P dd97849388ff46e8a6fe07a1c975df91: Received BeginTabletCopySession request for tablet 8cca4b0cda214a9c9451e74f1d7aa348 from peer 2b59a31f1cef4b1691b5f40f3fa2d9eb ({username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53421)
I20260926 14:47:20.755071 15363 tablet_copy_service.cc:161] P dd97849388ff46e8a6fe07a1c975df91: Beginning new tablet copy session on tablet 8cca4b0cda214a9c9451e74f1d7aa348 from peer 2b59a31f1cef4b1691b5f40f3fa2d9eb at {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53421: session id = 2b59a31f1cef4b1691b5f40f3fa2d9eb-8cca4b0cda214a9c9451e74f1d7aa348
I20260926 14:47:20.756275 15363 tablet_copy_source_session.cc:205] T 8cca4b0cda214a9c9451e74f1d7aa348 P dd97849388ff46e8a6fe07a1c975df91: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 14:47:20.760756 15679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cca4b0cda214a9c9451e74f1d7aa348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:20.767740 15679 tablet_copy_client.cc:791] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: tablet copy: Starting download of 0 data blocks...
I20260926 14:47:20.767935 15679 tablet_copy_client.cc:655] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: tablet copy: Starting download of 1 WAL segments...
I20260926 14:47:20.769802 15679 tablet_copy_client.cc:531] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 14:47:20.771679 15679 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Bootstrap starting.
I20260926 14:47:20.777330 15679 log.cc:824] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:20.784236 15679 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:47:20.784520 15679 tablet_bootstrap.cc:492] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Bootstrap complete.
I20260926 14:47:20.784825 15679 ts_tablet_manager.cc:1397] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260926 14:47:20.787221 15679 raft_consensus.cc:348] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:20.787492 15679 raft_consensus.cc:729] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b59a31f1cef4b1691b5f40f3fa2d9eb, State: Initialized, Role: FOLLOWER
I20260926 14:47:20.787928 15679 consensus_queue.cc:260] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b59a31f1cef4b1691b5f40f3fa2d9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38117 } } peers { permanent_uuid: "747dcdb586f74a30b7ac99f74601971c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40219 } } peers { permanent_uuid: "dd97849388ff46e8a6fe07a1c975df91" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41871 } }
I20260926 14:47:20.788872 15679 ts_tablet_manager.cc:1428] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260926 14:47:20.789155 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:20.788880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:51915)
0926 14:47:20.789011 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:20.789074 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 14:47:20.789623 15363 tablet_copy_service.cc:342] P dd97849388ff46e8a6fe07a1c975df91: Request end of tablet copy session 2b59a31f1cef4b1691b5f40f3fa2d9eb-8cca4b0cda214a9c9451e74f1d7aa348 received from {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53421
I20260926 14:47:20.789736 15363 tablet_copy_service.cc:433] P dd97849388ff46e8a6fe07a1c975df91: ending tablet copy session 2b59a31f1cef4b1691b5f40f3fa2d9eb-8cca4b0cda214a9c9451e74f1d7aa348 on tablet 8cca4b0cda214a9c9451e74f1d7aa348 with peer 2b59a31f1cef4b1691b5f40f3fa2d9eb
I20260926 14:47:20.789791 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:20.789580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:54275)
0926 14:47:20.789684 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:20.789730 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 14:47:21.137778 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:21.137351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:56647)
0926 14:47:21.137627 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:21.137679 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260926 14:47:21.137812 15612 raft_consensus.cc:1203] T 8cca4b0cda214a9c9451e74f1d7aa348 P 2b59a31f1cef4b1691b5f40f3fa2d9eb [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260926 14:47:21.138734 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:21.138517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:53219)
0926 14:47:21.138616 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:21.138671 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:47:21.201982 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:21.201549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:60101)
0926 14:47:21.201821 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:21.201885 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":66,"threads_started":1}
I20260926 14:47:22.139120 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:22.138621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:50919)
0926 14:47:22.138949 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:22.139019 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":76,"threads_started":1}
I20260926 14:47:22.139816 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:22.139417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:34455)
0926 14:47:22.139677 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:22.139729 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":64,"threads_started":1}
I20260926 14:47:22.203306 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:22.202820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:39133)
0926 14:47:22.203141 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:22.203206 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":76,"threads_started":1}
I20260926 14:47:23.140470 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:23.139986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:46167)
0926 14:47:23.140277 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:23.140349 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":94,"threads_started":1}
I20260926 14:47:23.140973 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:23.140559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:51329)
0926 14:47:23.140810 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:23.140871 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260926 14:47:23.204627 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:23.204141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:49271)
0926 14:47:23.204445 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:23.204515 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":77,"threads_started":1}
I20260926 14:47:24.142248 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:24.141729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:49361)
0926 14:47:24.142060 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:24.142133 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260926 14:47:24.142325 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:24.141819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:56103)
0926 14:47:24.142173 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:24.142239 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":56,"threads_started":1}
I20260926 14:47:24.143072 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:24.142815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:33653)
0926 14:47:24.142927 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:24.142976 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 14:47:24.205972 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:24.205525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:45575)
0926 14:47:24.205800 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:24.205873 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":81,"threads_started":1}
I20260926 14:47:25.143664 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:25.143148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:48247)
0926 14:47:25.143457 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:25.143535 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
I20260926 14:47:25.144150 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:25.143809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:39665)
0926 14:47:25.144011 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:25.144071 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260926 14:47:25.207152 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:25.206715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:35421)
0926 14:47:25.206983 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:25.207044 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":98,"threads_started":1}
I20260926 14:47:26.144914 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:26.144414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:36457)
0926 14:47:26.144724 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:26.144791 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":93,"threads_started":1}
I20260926 14:47:26.145205 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:26.144774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:53953)
0926 14:47:26.145041 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:26.145106 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
I20260926 14:47:26.208390 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:26.207926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:39327)
0926 14:47:26.208232 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:26.208294 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":78,"threads_started":1}
I20260926 14:47:27.146106 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:27.145644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:41675)
0926 14:47:27.145904 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:27.145974 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":71,"threads_started":1}
I20260926 14:47:27.146250 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:27.145869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:41757)
0926 14:47:27.146098 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:27.146152 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":78,"threads_started":1}
I20260926 14:47:27.209637 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:27.209189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:42365)
0926 14:47:27.209468 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:27.209527 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
I20260926 14:47:28.147333 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:28.146898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:53241)
0926 14:47:28.147161 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:28.147231 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":75,"threads_started":1}
I20260926 14:47:28.147398 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:28.146986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:33787)
0926 14:47:28.147252 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:28.147310 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":106,"threads_started":1}
I20260926 14:47:28.210855 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:28.210375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:33339)
0926 14:47:28.210642 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:28.210713 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260926 14:47:29.148599 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:29.148124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:52137)
0926 14:47:29.148444 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:29.148505 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":103,"threads_started":1}
I20260926 14:47:29.148622 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:29.148173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:45407)
0926 14:47:29.148484 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:29.148538 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":64,"threads_started":1}
I20260926 14:47:29.212220 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:29.211768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:38349)
0926 14:47:29.212068 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:29.212124 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":113,"threads_started":1}
I20260926 14:47:30.149760 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:30.149332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:45455)
0926 14:47:30.149596 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:30.149654 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":76,"threads_started":1}
I20260926 14:47:30.149766 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:30.149360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:54007)
0926 14:47:30.149622 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:30.149678 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":83,"threads_started":1}
I20260926 14:47:30.213451 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:30.213004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:35381)
0926 14:47:30.213280 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:30.213341 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":91,"threads_started":1}
I20260926 14:47:31.150983 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:31.150544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:38041)
0926 14:47:31.150824 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:31.150883 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260926 14:47:31.150987 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:31.150443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:57977)
0926 14:47:31.150823 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:31.150884 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":138,"threads_started":1}
I20260926 14:47:31.214596 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:31.214157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:35389)
0926 14:47:31.214431 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:31.214490 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":93,"threads_started":1}
I20260926 14:47:32.152158 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:32.151747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:47471)
0926 14:47:32.151998 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:32.152059 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":86,"threads_started":1}
I20260926 14:47:32.152217 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:32.151740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:35103)
0926 14:47:32.152082 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:32.152134 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":88,"threads_started":1}
I20260926 14:47:32.215835 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:32.215346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:43983)
0926 14:47:32.215635 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:32.215737 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":93,"threads_started":1}
I20260926 14:47:33.153303 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:33.152903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.3:38407)
0926 14:47:33.153146 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:33.153201 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":70,"threads_started":1}
I20260926 14:47:33.153309 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:33.152922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.1:54757)
0926 14:47:33.153160 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:33.153222 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":80,"threads_started":1}
I20260926 14:47:33.216992 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:33.216568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45781 (local address 127.0.98.2:58587)
0926 14:47:33.216831 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:33.216891 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:45781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":76,"threads_started":1}
I20260926 14:47:33.683429 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 15225
I20260926 14:47:33.688898 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 15392
I20260926 14:47:33.694082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 15528
2026-09-26T14:47:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19022 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:47:33 dist-test-slave-s7k6 krb5kdc[15727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:47:33 dist-test-slave-s7k6 krb5kdc[15727](info): set up 2 sockets
Sep 26 14:47:33 dist-test-slave-s7k6 krb5kdc[15727](info): commencing operation
krb5kdc: starting...
W20260926 14:47:34.658670 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:47:34 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:47:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:47:34.773864 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35535
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:34.884596 15740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:34.884886 15740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:34.884925 15740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:34.888957 15740 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:47:34.889012 15740 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:47:34.889027 15740 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:47:34.889039 15740 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:47:34.889055 15740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:34.889071 15740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:34.889091 15740 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:47:34.889108 15740 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:47:34.894452 15740 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:34.895555 15740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:34.896550 15740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:34.905264 15746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:34.905237 15748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:34.905237 15745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:34.905303 15740 server_base.cc:1034] running on GCE node
I20260926 14:47:34.906129 15740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:34.907311 15740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:34.908492 15740 hybrid_clock.cc:648] HybridClock initialized: now 1790434054908472 us; error 39 us; skew 500 ppm
Sep 26 14:47:34 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434054, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:34.911321 15740 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:47:34.912225 15740 webserver.cc:460] Webserver started at http://127.0.98.62:35509/ using document root <none> and password file <none>
I20260926 14:47:34.912762 15740 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:34.912823 15740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:34.913025 15740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:34.915127 15740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "9dfcbc0210034697b346ed5d0d11749c"
format_stamp: "Formatted at 2026-09-26 14:47:34 on dist-test-slave-s7k6"
server_key: "e43142b99cc4c3f7bb61e5bd824e878b"
server_key_iv: "436815f1f0e2e50640d4af1e10061534"
server_key_version: "encryptionkey@0"
I20260926 14:47:34.915652 15740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "9dfcbc0210034697b346ed5d0d11749c"
format_stamp: "Formatted at 2026-09-26 14:47:34 on dist-test-slave-s7k6"
server_key: "e43142b99cc4c3f7bb61e5bd824e878b"
server_key_iv: "436815f1f0e2e50640d4af1e10061534"
server_key_version: "encryptionkey@0"
I20260926 14:47:34.919431 15740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:47:34.921871 15755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:34.923139 15740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:47:34.923249 15740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "9dfcbc0210034697b346ed5d0d11749c"
format_stamp: "Formatted at 2026-09-26 14:47:34 on dist-test-slave-s7k6"
server_key: "e43142b99cc4c3f7bb61e5bd824e878b"
server_key_iv: "436815f1f0e2e50640d4af1e10061534"
server_key_version: "encryptionkey@0"
I20260926 14:47:34.923360 15740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:34.936440 15740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:34.939344 15740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:34.939527 15740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:34.946079 15740 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35535
I20260926 14:47:34.946342 15807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35535 every 8 connection(s)
I20260926 14:47:34.947268 15740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:47:34.949780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 15740
I20260926 14:47:34.949896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:47:34.950091 15808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:34.950201 392 external_mini_cluster.cc:1428] Setting key ce1b6893b6eee9dd914bcf97a864ada1
I20260926 14:47:34.956995 15808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Bootstrap starting.
Sep 26 14:47:34 dist-test-slave-s7k6 krb5kdc[15727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:34.959408 15808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:34.960265 15808 log.cc:824] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:34.962352 15808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: No bootstrap required, opened a new log
I20260926 14:47:34.964591 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:34.951555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44696 (local address 127.0.98.62:35535)
0926 14:47:34.951944 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:34.951966 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:34.952002 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:47:34.952705 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:34.952724 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:34.953111 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:34.953397 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:34.954280 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:34.955406 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:34.956564 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:34.957078 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:34.957224 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:34.959936 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:34.959943 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:34.959959 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:34.959991 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:34.961688 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:34.962240 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:34.962245 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:34.962250 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:34.962363 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:34.962662 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:34.962666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:34.962667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:34.963107 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:34.963287 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:34.963581 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 14:47:34.963857 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":112,"threads_started":1}
I20260926 14:47:34.965974 15808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:47:34.966202 15808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:34.966260 15808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfcbc0210034697b346ed5d0d11749c, State: Initialized, Role: FOLLOWER
I20260926 14:47:34.966681 15808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:47:34.966866 15808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:47:34.966919 15808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:47:34.966988 15808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:34.967823 15808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:47:34.968190 15808 leader_election.cc:302] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dfcbc0210034697b346ed5d0d11749c; no voters:
I20260926 14:47:34.968523 15808 leader_election.cc:288] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:47:34.968618 15813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:47:34.968827 15813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 1 LEADER]: Becoming Leader. State: Replica: 9dfcbc0210034697b346ed5d0d11749c, State: Running, Role: LEADER
I20260926 14:47:34.969179 15813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:47:34.969743 15808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:47:34.970846 15814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dfcbc0210034697b346ed5d0d11749c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } } }
I20260926 14:47:34.970994 15814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:34.971271 15815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dfcbc0210034697b346ed5d0d11749c. Latest consensus state: current_term: 1 leader_uuid: "9dfcbc0210034697b346ed5d0d11749c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } } }
I20260926 14:47:34.971354 15815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:34.971374 15822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:47:34.974488 15822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:47:34.980877 15822 catalog_manager.cc:1347] Generated new cluster ID: a75a1021a5804640b182b744a6832c8a
I20260926 14:47:34.980983 15822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:47:34.996063 15822 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:47:34.996989 15822 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:47:35.000641 15822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Generated new TSK 0
I20260926 14:47:35.001292 15822 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:47:35.033977 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35535
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:35.147275 15834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:35.147545 15834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:35.147588 15834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:35.151660 15834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:35.151726 15834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:35.151784 15834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:47:35.157083 15834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:35.158105 15834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:35.159104 15834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:35.168982 15839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:35.169075 15834 server_base.cc:1034] running on GCE node
W20260926 14:47:35.169008 15842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:35.169045 15840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:35.169677 15834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:35.170471 15834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:35.171650 15834 hybrid_clock.cc:648] HybridClock initialized: now 1790434055171617 us; error 52 us; skew 500 ppm
Sep 26 14:47:35 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:35.174520 15834 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:47:35.175463 15834 webserver.cc:460] Webserver started at http://127.0.98.1:33603/ using document root <none> and password file <none>
I20260926 14:47:35.176014 15834 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:35.176080 15834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:35.176249 15834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:35.178334 15834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "0bfd944b41d952bc0ecd0000648ecc50"
server_key_iv: "62057d55f6acdbbbc2b5c0ba393b255a"
server_key_version: "encryptionkey@0"
I20260926 14:47:35.178905 15834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "0bfd944b41d952bc0ecd0000648ecc50"
server_key_iv: "62057d55f6acdbbbc2b5c0ba393b255a"
server_key_version: "encryptionkey@0"
I20260926 14:47:35.182873 15834 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:47:35.185571 15849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:35.186971 15834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:47:35.187134 15834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "0bfd944b41d952bc0ecd0000648ecc50"
server_key_iv: "62057d55f6acdbbbc2b5c0ba393b255a"
server_key_version: "encryptionkey@0"
I20260926 14:47:35.187242 15834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:35.206521 15834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:35.209841 15834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:35.210052 15834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:35.210649 15834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:35.211516 15834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:35.211565 15834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:35.211618 15834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:35.211644 15834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:35.220330 15834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43843
I20260926 14:47:35.220359 15962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43843 every 8 connection(s)
I20260926 14:47:35.221227 15834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:47:35.230492 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 15834
I20260926 14:47:35.230610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:47:35.230912 392 external_mini_cluster.cc:1428] Setting key 21d7be616bf3789624e72a2a4ea4e67a
Sep 26 14:47:35 dist-test-slave-s7k6 krb5kdc[15727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:35.235695 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.223236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36761 (local address 127.0.98.62:35535)
0926 14:47:35.223398 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:35.223402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:35.224509 (+ 1107us) server_negotiation.cc:407] Connection header received
0926 14:47:35.225404 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:35.225407 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:35.225456 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:35.225558 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:35.226877 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.227518 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.228688 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.228929 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.229015 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.231571 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:35.231575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:35.231577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:35.231607 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:35.233228 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:35.233825 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:35.233828 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:35.233830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:35.233923 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:35.234485 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:35.234487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:35.234488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:35.234675 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:35.234787 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:35.235449 (+ 662us) server_negotiation.cc:299] Negotiation successful
0926 14:47:35.235570 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 14:47:35.236631 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.223632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:36761)
0926 14:47:35.224334 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:35.224380 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:35.225187 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:35.225698 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:35.225705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:35.226070 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:35.226686 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.226704 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.227632 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.227634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.228564 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.228574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.229067 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.229070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.229220 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.229819 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:35.229836 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:35.231399 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:35.233370 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:35.233375 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:35.233388 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:35.233655 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:35.234088 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:35.234092 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:35.234094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:35.234386 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:35.234790 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:35.234798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:35.235149 (+ 351us) client_negotiation.cc:769] Sending connection context
0926 14:47:35.235371 (+ 222us) client_negotiation.cc:240] Negotiation successful
0926 14:47:35.235663 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":129,"threads_started":1}
I20260926 14:47:35.237950 15963 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35535
I20260926 14:47:35.238191 15963 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:35.238686 15963 heartbeater.cc:511] Master 127.0.98.62:35535 requested a full tablet report, sending...
I20260926 14:47:35.240459 15772 ts_manager.cc:194] Registered new tserver with Master: 9ae3d5b99ad84ef3b7d53e1062932d6f (127.0.98.1:43843)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 14:47:35.241814 15772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36761
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:47:35.257588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35535
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:35.372031 15968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:35.372308 15968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:35.372351 15968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:35.376565 15968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:35.376636 15968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:35.376698 15968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:47:35.382200 15968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:35.383361 15968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:35.384402 15968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:35.393864 15976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:35.393934 15973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:35.393954 15974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:35.394310 15968 server_base.cc:1034] running on GCE node
I20260926 14:47:35.394703 15968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:35.395361 15968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:35.396531 15968 hybrid_clock.cc:648] HybridClock initialized: now 1790434055396502 us; error 40 us; skew 500 ppm
Sep 26 14:47:35 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:35.399362 15968 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:47:35.400315 15968 webserver.cc:460] Webserver started at http://127.0.98.2:34883/ using document root <none> and password file <none>
I20260926 14:47:35.400914 15968 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:35.400986 15968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:35.401196 15968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:35.403417 15968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "7361b1493e19456896c3d5648556d40f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "e13f55c4d32414ef0e073c44d257f614"
server_key_iv: "790e1e314bdacb9bfa738dd045cdff18"
server_key_version: "encryptionkey@0"
I20260926 14:47:35.403977 15968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7361b1493e19456896c3d5648556d40f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "e13f55c4d32414ef0e073c44d257f614"
server_key_iv: "790e1e314bdacb9bfa738dd045cdff18"
server_key_version: "encryptionkey@0"
I20260926 14:47:35.407912 15968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:47:35.410697 15983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:35.412213 15968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:47:35.412339 15968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "7361b1493e19456896c3d5648556d40f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "e13f55c4d32414ef0e073c44d257f614"
server_key_iv: "790e1e314bdacb9bfa738dd045cdff18"
server_key_version: "encryptionkey@0"
I20260926 14:47:35.412470 15968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:35.439854 15968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:35.443351 15968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:35.443534 15968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:35.444146 15968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:35.444983 15968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:35.445029 15968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:35.445091 15968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:35.445114 15968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:35.453572 15968 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40841
I20260926 14:47:35.453683 16096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40841 every 8 connection(s)
I20260926 14:47:35.454514 15968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:47:35.464214 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 15968
I20260926 14:47:35.464330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
Sep 26 14:47:35 dist-test-slave-s7k6 krb5kdc[15727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:35.464641 392 external_mini_cluster.cc:1428] Setting key cb157feef90e3ec5242d166ef87ddc3e
I20260926 14:47:35.469805 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.456444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43519 (local address 127.0.98.62:35535)
0926 14:47:35.456605 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:35.456608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:35.457652 (+ 1044us) server_negotiation.cc:407] Connection header received
0926 14:47:35.458815 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:35.458819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:35.458893 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:35.458967 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:35.460283 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.460826 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.462079 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.462317 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.462408 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.465286 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:35.465290 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:35.465292 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:35.465321 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:35.467210 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:35.467852 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:35.467856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:35.467858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:35.467954 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:35.468446 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:35.468449 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:35.468450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:35.468664 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:35.468747 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:35.469474 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 14:47:35.469635 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:47:35.470904 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.456810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:43519)
0926 14:47:35.457471 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:35.457515 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:35.458522 (+ 1007us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:35.459166 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:35.459174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:35.459556 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:35.460107 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.460127 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.460952 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.460955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.461944 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.461956 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.462466 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.462470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.462650 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.463262 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:35.463279 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:35.465038 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:35.467360 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:35.467365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:35.467380 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:35.467750 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:35.468056 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:35.468060 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:35.468061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:35.468339 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:35.468761 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:35.468768 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:35.469148 (+ 380us) client_negotiation.cc:769] Sending connection context
0926 14:47:35.469378 (+ 230us) client_negotiation.cc:240] Negotiation successful
0926 14:47:35.469746 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":126,"threads_started":1}
I20260926 14:47:35.472306 16097 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35535
I20260926 14:47:35.472570 16097 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:35.473110 16097 heartbeater.cc:511] Master 127.0.98.62:35535 requested a full tablet report, sending...
I20260926 14:47:35.474287 15772 ts_manager.cc:194] Registered new tserver with Master: 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 14:47:35.475081 15772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43519
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:47:35.494082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35535
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:35.607332 16102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:35.607617 16102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:35.607661 16102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:35.611840 16102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:35.611914 16102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:35.611976 16102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:47:35.617663 16102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:35.618777 16102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:35.619849 16102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:35.629594 16110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:35.629658 16108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:35.629594 16107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:35.629817 16102 server_base.cc:1034] running on GCE node
I20260926 14:47:35.630368 16102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:35.631104 16102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:35.632292 16102 hybrid_clock.cc:648] HybridClock initialized: now 1790434055632260 us; error 45 us; skew 500 ppm
Sep 26 14:47:35 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:35.635116 16102 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:47:35.636018 16102 webserver.cc:460] Webserver started at http://127.0.98.3:44331/ using document root <none> and password file <none>
I20260926 14:47:35.636550 16102 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:35.636617 16102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:35.636796 16102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:47:35.638985 16102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "0f772a2f91c34cd494c6dae3a9faf35f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "27ce6ba81de7043f645ff16adf8c501b"
server_key_iv: "c2bc914ca48781226e2af2babe5ba028"
server_key_version: "encryptionkey@0"
I20260926 14:47:35.639562 16102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0f772a2f91c34cd494c6dae3a9faf35f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "27ce6ba81de7043f645ff16adf8c501b"
server_key_iv: "c2bc914ca48781226e2af2babe5ba028"
server_key_version: "encryptionkey@0"
I20260926 14:47:35.643525 16102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:47:35.646124 16117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:35.647485 16102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:47:35.647632 16102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "0f772a2f91c34cd494c6dae3a9faf35f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "27ce6ba81de7043f645ff16adf8c501b"
server_key_iv: "c2bc914ca48781226e2af2babe5ba028"
server_key_version: "encryptionkey@0"
I20260926 14:47:35.647748 16102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:35.668423 16102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:35.671888 16102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:35.672081 16102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:35.672693 16102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:35.673540 16102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:35.673590 16102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:35.673704 16102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:35.673738 16102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:35.682548 16102 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41949
I20260926 14:47:35.682576 16230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41949 every 8 connection(s)
I20260926 14:47:35.683564 16102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:47:35.690725 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 16102
I20260926 14:47:35.690886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:47:35.691193 392 external_mini_cluster.cc:1428] Setting key 0de4418237cd2e154e75db40f5a67a31
Sep 26 14:47:35 dist-test-slave-s7k6 krb5kdc[15727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:35.697477 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.685497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52743 (local address 127.0.98.62:35535)
0926 14:47:35.685635 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:35.685639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:35.686689 (+ 1050us) server_negotiation.cc:407] Connection header received
0926 14:47:35.687576 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:35.687579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:35.687642 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:35.687716 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:35.688977 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.689543 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.690799 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.691115 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.691222 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.694045 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:35.694049 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:35.694051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:35.694076 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:35.695422 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:35.695945 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:35.695947 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:35.695949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:35.696026 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:35.696405 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:35.696408 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:35.696409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:35.696609 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:35.696711 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:35.697236 (+ 525us) server_negotiation.cc:299] Negotiation successful
0926 14:47:35.697354 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:47:35.698336 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.685889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:52743)
0926 14:47:35.686529 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:35.686574 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:35.687397 (+ 823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:35.687896 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:35.687905 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:35.688324 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:35.688830 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.688843 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.689677 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.689680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.690647 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.690657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.691268 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.691272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.691439 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.692179 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:35.692207 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:35.693887 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:35.695545 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:35.695552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:35.695565 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:35.695861 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:35.696132 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:35.696135 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:35.696138 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:35.696287 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:35.696720 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:35.696726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:35.696979 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 14:47:35.697151 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 14:47:35.697438 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":120,"threads_started":1}
I20260926 14:47:35.699528 16231 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35535
I20260926 14:47:35.699779 16231 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:35.700289 16231 heartbeater.cc:511] Master 127.0.98.62:35535 requested a full tablet report, sending...
I20260926 14:47:35.701336 15772 ts_manager.cc:194] Registered new tserver with Master: 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949)
I20260926 14:47:35.701864 15772 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52743
I20260926 14:47:35.705438 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:47:35.718253 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.707369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44712 (local address 127.0.98.62:35535)
0926 14:47:35.707500 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:35.707503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:35.707630 (+ 127us) server_negotiation.cc:407] Connection header received
0926 14:47:35.707721 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:35.707723 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:35.707760 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:35.707825 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:35.708506 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.709000 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.709978 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.710217 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.710297 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.715286 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:35.715289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:35.715291 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:35.715316 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:35.716676 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:35.717070 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:35.717073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:35.717074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:35.717156 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:35.717395 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:35.717398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:35.717400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:35.717622 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:35.717741 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:35.717983 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 14:47:35.718093 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:47:35.721065 15772 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:47:35.724329 15772 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:47:35.738981 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.732869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41949 (local address 127.0.0.1:49526)
0926 14:47:35.733249 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:35.733307 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:35.733439 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:35.733986 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:35.733992 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:35.734013 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:35.734178 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.734185 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.735807 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.735811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.737164 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.737174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.738318 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.738323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.738456 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.738544 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 14:47:35.738651 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 14:47:35.738824 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":28,"thread_start_us":100,"threads_started":1}
I20260926 14:47:35.739140 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.733129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43843 (local address 127.0.0.1:34100)
0926 14:47:35.733809 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:35.733822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:35.733902 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:35.734915 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:35.734919 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:35.734936 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:35.735044 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.735050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.736197 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.736200 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.737534 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.737545 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.738365 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.738369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.738491 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.738744 (+ 253us) client_negotiation.cc:769] Sending connection context
0926 14:47:35.738845 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 14:47:35.738940 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":348,"thread_start_us":86,"threads_started":1}
I20260926 14:47:35.739297 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.733643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40841 (local address 127.0.0.1:50430)
0926 14:47:35.734074 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:35.734086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:35.734155 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:35.735048 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:35.735051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:35.735065 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:35.735140 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.735146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.736392 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.736395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.737540 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.737546 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.738683 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.738687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.738798 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.738822 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:47:35.738862 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 14:47:35.738938 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":8,"thread_start_us":85,"threads_started":1}
I20260926 14:47:35.739395 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.733011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49526 (local address 127.0.98.3:41949)
0926 14:47:35.733471 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:35.733483 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:35.733509 (+ 26us) server_negotiation.cc:407] Connection header received
0926 14:47:35.733583 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:35.733588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:35.733816 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:35.734332 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:35.734495 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.735668 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.737302 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.738062 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.738333 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.738493 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:35.738834 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 14:47:35.738939 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":123,"threads_started":1}
I20260926 14:47:35.739497 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.734189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.0.98.2:40841)
0926 14:47:35.734613 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:35.734621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:35.734641 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:47:35.734705 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:35.734708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:35.734906 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:35.735257 (+ 351us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:35.735377 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.736264 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.737690 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.738544 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.739057 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.739202 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:35.739253 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 14:47:35.739360 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":137,"threads_started":1}
I20260926 14:47:35.740602 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.733758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.0.98.1:43843)
0926 14:47:35.734400 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:35.734412 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:35.734436 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:47:35.734516 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:35.734521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:35.734727 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:35.734913 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:35.735150 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.736079 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.737699 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.738236 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.738556 (+ 320us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.738718 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:35.740228 (+ 1510us) server_negotiation.cc:299] Negotiation successful
0926 14:47:35.740489 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":59,"threads_started":1}
I20260926 14:47:35.742542 16165 tablet_service.cc:1467] Processing CreateTablet for tablet caa4b96740c9418490cf96dd52378807 (DEFAULT_TABLE table=test-table [id=e998b74fdd7f4a0bb3d64725b0498130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:35.742528 16031 tablet_service.cc:1467] Processing CreateTablet for tablet caa4b96740c9418490cf96dd52378807 (DEFAULT_TABLE table=test-table [id=e998b74fdd7f4a0bb3d64725b0498130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:35.743697 16031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa4b96740c9418490cf96dd52378807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:35.743702 16165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa4b96740c9418490cf96dd52378807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:35.743340 15897 tablet_service.cc:1467] Processing CreateTablet for tablet caa4b96740c9418490cf96dd52378807 (DEFAULT_TABLE table=test-table [id=e998b74fdd7f4a0bb3d64725b0498130]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:35.744371 15897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa4b96740c9418490cf96dd52378807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:35.750347 16248 tablet_bootstrap.cc:492] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Bootstrap starting.
I20260926 14:47:35.751412 16249 tablet_bootstrap.cc:492] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: Bootstrap starting.
I20260926 14:47:35.752179 16250 tablet_bootstrap.cc:492] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: Bootstrap starting.
I20260926 14:47:35.752902 16248 tablet_bootstrap.cc:654] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:35.753791 16248 log.cc:824] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:35.753994 16250 tablet_bootstrap.cc:654] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:35.754108 16249 tablet_bootstrap.cc:654] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:35.755020 16249 log.cc:824] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:35.755162 16250 log.cc:824] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:35.756623 16248 tablet_bootstrap.cc:492] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: No bootstrap required, opened a new log
I20260926 14:47:35.756835 16248 ts_tablet_manager.cc:1397] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260926 14:47:35.756855 16250 tablet_bootstrap.cc:492] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: No bootstrap required, opened a new log
I20260926 14:47:35.756960 16249 tablet_bootstrap.cc:492] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: No bootstrap required, opened a new log
I20260926 14:47:35.756989 16250 ts_tablet_manager.cc:1397] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:47:35.757118 16249 ts_tablet_manager.cc:1397] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 14:47:35.759960 16250 raft_consensus.cc:348] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:35.760305 16250 raft_consensus.cc:374] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:35.760390 16250 raft_consensus.cc:729] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f772a2f91c34cd494c6dae3a9faf35f, State: Initialized, Role: FOLLOWER
I20260926 14:47:35.760843 16250 consensus_queue.cc:260] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:35.760812 16249 raft_consensus.cc:348] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:35.761137 16249 raft_consensus.cc:374] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:35.761221 16249 raft_consensus.cc:729] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7361b1493e19456896c3d5648556d40f, State: Initialized, Role: FOLLOWER
I20260926 14:47:35.761539 16231 heartbeater.cc:503] Master 127.0.98.62:35535 was elected leader, sending a full tablet report...
I20260926 14:47:35.761703 16249 consensus_queue.cc:260] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:35.761899 16250 ts_tablet_manager.cc:1428] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:47:35.762585 16097 heartbeater.cc:503] Master 127.0.98.62:35535 was elected leader, sending a full tablet report...
I20260926 14:47:35.762715 16249 ts_tablet_manager.cc:1428] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:47:35.763715 16248 raft_consensus.cc:348] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:35.763947 16248 raft_consensus.cc:374] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:35.764002 16248 raft_consensus.cc:729] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae3d5b99ad84ef3b7d53e1062932d6f, State: Initialized, Role: FOLLOWER
I20260926 14:47:35.764420 16248 consensus_queue.cc:260] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:35.765381 16248 ts_tablet_manager.cc:1428] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260926 14:47:35.765405 15963 heartbeater.cc:503] Master 127.0.98.62:35535 was elected leader, sending a full tablet report...
I20260926 14:47:35.902482 16254 raft_consensus.cc:482] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:47:35.902786 16254 raft_consensus.cc:504] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:35.904012 16254 leader_election.cc:288] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841), 9ae3d5b99ad84ef3b7d53e1062932d6f (127.0.98.1:43843)
I20260926 14:47:35.907904 16233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.904322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43843 (local address 127.0.98.3:47415)
0926 14:47:35.904469 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:35.904484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:35.904586 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:35.904842 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:35.904845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:35.904863 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:35.904991 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.904997 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.905757 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.905760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.906635 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.906646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.907490 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.907494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.907597 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.907614 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:47:35.907683 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:47:35.907762 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"spinlock_wait_cycles":27264}
I20260926 14:47:35.907904 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.904426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47415 (local address 127.0.98.1:43843)
0926 14:47:35.904551 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:35.904555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:35.904569 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:47:35.904703 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:35.904706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:35.904750 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:35.904822 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:35.905095 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.905649 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.906789 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.907371 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.907471 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.907540 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:35.907707 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 14:47:35.907781 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:47:35.908115 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.904423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38765 (local address 127.0.98.2:40841)
0926 14:47:35.904543 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:35.904547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:35.904890 (+ 343us) server_negotiation.cc:407] Connection header received
0926 14:47:35.905048 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:35.905051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:35.905100 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:35.905169 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:35.905428 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.905937 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.907171 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.907594 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.907693 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.907757 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:35.907926 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 14:47:35.908002 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:47:35.908133 16257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.904323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40841 (local address 127.0.98.3:38765)
0926 14:47:35.904797 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:35.904808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:35.904932 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:35.905184 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:35.905187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:35.905202 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:35.905315 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.905320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.906087 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.906090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.906999 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:35.907018 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.907710 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.907714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:35.907815 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.907838 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:47:35.907902 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:47:35.907972 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":156,"spinlock_wait_cycles":768,"thread_start_us":131,"threads_started":1}
I20260926 14:47:35.908663 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4b96740c9418490cf96dd52378807" candidate_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" is_pre_election: true
I20260926 14:47:35.908748 16051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4b96740c9418490cf96dd52378807" candidate_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7361b1493e19456896c3d5648556d40f" is_pre_election: true
I20260926 14:47:35.908977 15917 raft_consensus.cc:2454] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f772a2f91c34cd494c6dae3a9faf35f in term 0.
I20260926 14:47:35.909040 16051 raft_consensus.cc:2454] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f772a2f91c34cd494c6dae3a9faf35f in term 0.
I20260926 14:47:35.909523 16121 leader_election.cc:302] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f772a2f91c34cd494c6dae3a9faf35f, 9ae3d5b99ad84ef3b7d53e1062932d6f; no voters:
I20260926 14:47:35.909811 16254 raft_consensus.cc:2781] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:47:35.909901 16254 raft_consensus.cc:482] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:47:35.909940 16254 raft_consensus.cc:3037] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:35.911034 16254 raft_consensus.cc:504] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:35.911417 16254 leader_election.cc:288] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [CANDIDATE]: Term 1 election: Requested vote from peers 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841), 9ae3d5b99ad84ef3b7d53e1062932d6f (127.0.98.1:43843)
I20260926 14:47:35.911757 16051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4b96740c9418490cf96dd52378807" candidate_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7361b1493e19456896c3d5648556d40f"
I20260926 14:47:35.911793 15917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4b96740c9418490cf96dd52378807" candidate_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f"
I20260926 14:47:35.911883 16051 raft_consensus.cc:3037] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:35.911902 15917 raft_consensus.cc:3037] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:35.912907 15917 raft_consensus.cc:2454] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f772a2f91c34cd494c6dae3a9faf35f in term 1.
I20260926 14:47:35.912923 16051 raft_consensus.cc:2454] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f772a2f91c34cd494c6dae3a9faf35f in term 1.
I20260926 14:47:35.913209 16121 leader_election.cc:302] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f772a2f91c34cd494c6dae3a9faf35f, 9ae3d5b99ad84ef3b7d53e1062932d6f; no voters:
I20260926 14:47:35.913374 16254 raft_consensus.cc:2781] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:47:35.913609 16254 raft_consensus.cc:686] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 1 LEADER]: Becoming Leader. State: Replica: 0f772a2f91c34cd494c6dae3a9faf35f, State: Running, Role: LEADER
I20260926 14:47:35.913967 16254 consensus_queue.cc:237] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:35.917277 15771 catalog_manager.cc:5495] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } health_report { overall_health: UNKNOWN } } }
I20260926 14:47:35.927825 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.924404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49542 (local address 127.0.98.3:41949)
0926 14:47:35.924518 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:35.924521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:35.924534 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:47:35.924664 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:35.924667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:35.924721 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:35.924809 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:47:35.924993 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.925506 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.926385 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:35.926614 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:35.926704 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:35.926989 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:47:35.927455 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:47:35.927548 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:35.927623 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 14:47:35.927691 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:47:35.933670 15917 raft_consensus.cc:1261] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Refusing update from remote peer 0f772a2f91c34cd494c6dae3a9faf35f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:35.933670 16051 raft_consensus.cc:1261] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f [term 1 FOLLOWER]: Refusing update from remote peer 0f772a2f91c34cd494c6dae3a9faf35f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:35.934348 16254 consensus_queue.cc:1035] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:47:35.934449 16232 tablet.cc:2369] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:47:35.934633 16258 consensus_queue.cc:1035] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:47:35.945267 16262 mvcc.cc:205] Tried to move back new op lower bound from 7333617893096992768 to 7333617893026197504. Current Snapshot: MvccSnapshot[applied={T|T < 7333617893096992768}]
I20260926 14:47:35.947894 16263 mvcc.cc:205] Tried to move back new op lower bound from 7333617893096992768 to 7333617893026197504. Current Snapshot: MvccSnapshot[applied={T|T < 7333617893096992768}]
I20260926 14:47:35.963023 15772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:47:35.963203 15772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:47:35.965803 15772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Sending DeleteTablet for 3 replicas of tablet caa4b96740c9418490cf96dd52378807
I20260926 14:47:35.966706 15897 tablet_service.cc:1514] Processing DeleteTablet for tablet caa4b96740c9418490cf96dd52378807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34100
I20260926 14:47:35.966806 16165 tablet_service.cc:1514] Processing DeleteTablet for tablet caa4b96740c9418490cf96dd52378807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49526
I20260926 14:47:35.967271 16273 tablet_replica.cc:330] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: stopping tablet replica
I20260926 14:47:35.967299 16031 tablet_service.cc:1514] Processing DeleteTablet for tablet caa4b96740c9418490cf96dd52378807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50430
I20260926 14:47:35.967593 16273 raft_consensus.cc:2229] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:47:35.967979 16272 tablet_replica.cc:330] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: stopping tablet replica
I20260926 14:47:35.968039 16273 raft_consensus.cc:2258] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:35.968251 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 15834
I20260926 14:47:35.968329 16272 raft_consensus.cc:2229] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:47:35.968415 16274 tablet_replica.cc:330] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: stopping tablet replica
I20260926 14:47:35.968657 16272 raft_consensus.cc:2258] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:35.968693 16274 raft_consensus.cc:2229] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:47:35.968993 16274 raft_consensus.cc:2258] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:35.970695 16273 ts_tablet_manager.cc:1905] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:35.974238 16273 ts_tablet_manager.cc:1918] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:47:35.974486 16273 log.cc:1196] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/caa4b96740c9418490cf96dd52378807
I20260926 14:47:35.974886 16273 ts_tablet_manager.cc:1939] T caa4b96740c9418490cf96dd52378807 P 0f772a2f91c34cd494c6dae3a9faf35f: Deleting consensus metadata
I20260926 14:47:35.976154 16274 ts_tablet_manager.cc:1905] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:35.978056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 15968
W20260926 14:47:35.978153 15759 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43843 (error 108)
I20260926 14:47:35.980090 15756 catalog_manager.cc:4841] TS 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949): tablet caa4b96740c9418490cf96dd52378807 (table test-table [id=e998b74fdd7f4a0bb3d64725b0498130]) successfully deleted
I20260926 14:47:35.984598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 16102
I20260926 14:47:35.986455 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.984329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43843 (local address 127.0.0.1:34110)
0926 14:47:35.984539 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:35.984645 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 14:47:35.986662 15759 catalog_manager.cc:4584] TS 9ae3d5b99ad84ef3b7d53e1062932d6f (127.0.98.1:43843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet caa4b96740c9418490cf96dd52378807: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43843: connect: Connection refused (error 111)
I20260926 14:47:35.987723 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:35.987342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40841 (local address 127.0.0.1:50444)
0926 14:47:35.987486 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:35.987577 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 14:47:35.991762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 15740
I20260926 14:47:55.997794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35535
--webserver_interface=127.0.98.62
--webserver_port=35509
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:47:56.118849 16276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:56.119150 16276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:56.119210 16276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:56.123458 16276 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:47:56.123517 16276 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:47:56.123538 16276 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:47:56.123554 16276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:47:56.123580 16276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:56.123613 16276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:56.123651 16276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:47:56.123680 16276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:47:56.129245 16276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:56.130414 16276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:56.131551 16276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:56.145434 16285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:56.145669 16276 server_base.cc:1034] running on GCE node
W20260926 14:47:56.145448 16283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:56.145448 16282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:56.146564 16276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:56.147837 16276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:56.149027 16276 hybrid_clock.cc:648] HybridClock initialized: now 1790434076148979 us; error 65 us; skew 500 ppm
Sep 26 14:47:56 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:56.152154 16276 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:47:56.153249 16276 webserver.cc:460] Webserver started at http://127.0.98.62:35509/ using document root <none> and password file <none>
I20260926 14:47:56.153863 16276 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:56.153949 16276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:56.158161 16276 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:47:56.160565 16292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:56.161895 16276 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:47:56.162019 16276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "9dfcbc0210034697b346ed5d0d11749c"
format_stamp: "Formatted at 2026-09-26 14:47:34 on dist-test-slave-s7k6"
server_key: "e43142b99cc4c3f7bb61e5bd824e878b"
server_key_iv: "436815f1f0e2e50640d4af1e10061534"
server_key_version: "encryptionkey@0"
I20260926 14:47:56.162655 16276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:56.191323 16276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:56.194459 16276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:56.194654 16276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:56.201750 16276 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35535
I20260926 14:47:56.202028 16344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35535 every 8 connection(s)
I20260926 14:47:56.202941 16276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:47:56.204686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 16276
I20260926 14:47:56.205170 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43843
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35535
--webserver_port=33603
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:56.208353 16345 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:47:56.210132 16345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Bootstrap starting.
I20260926 14:47:56.227550 16345 log.cc:824] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:56.236541 16345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:47:56.236913 16345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Bootstrap complete.
I20260926 14:47:56.240931 16345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:47:56.241331 16345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfcbc0210034697b346ed5d0d11749c, State: Initialized, Role: FOLLOWER
I20260926 14:47:56.241808 16345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:47:56.241923 16345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:47:56.241989 16345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:47:56.242080 16345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 1 FOLLOWER]: Advancing to term 2
I20260926 14:47:56.244094 16345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:47:56.244414 16345 leader_election.cc:302] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dfcbc0210034697b346ed5d0d11749c; no voters:
I20260926 14:47:56.244716 16345 leader_election.cc:288] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 14:47:56.244814 16349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 2 FOLLOWER]: Leader election won for term 2
I20260926 14:47:56.245015 16349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 2 LEADER]: Becoming Leader. State: Replica: 9dfcbc0210034697b346ed5d0d11749c, State: Running, Role: LEADER
I20260926 14:47:56.245321 16349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:47:56.245976 16345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:47:56.246300 16349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dfcbc0210034697b346ed5d0d11749c. Latest consensus state: current_term: 2 leader_uuid: "9dfcbc0210034697b346ed5d0d11749c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } } }
I20260926 14:47:56.246528 16349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:56.247749 16353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:47:56.247859 16350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9dfcbc0210034697b346ed5d0d11749c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } } }
I20260926 14:47:56.247956 16350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: This master's current role is: LEADER
I20260926 14:47:56.252607 16353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:47:56.253229 16353 catalog_manager.cc:1259] Loaded cluster ID: a75a1021a5804640b182b744a6832c8a
I20260926 14:47:56.253293 16353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:47:56.255739 16353 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:47:56.256445 16353 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Loaded TSK: 0
I20260926 14:47:56.274096 16353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Generated new TSK 1
I20260926 14:47:56.274406 16353 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:47:56.325918 16347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:56.326210 16347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:56.326256 16347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:56.330397 16347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:56.330482 16347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:56.330545 16347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:47:56.336158 16347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33603
--tserver_master_addrs=127.0.98.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:56.337239 16347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:56.338284 16347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:56.348166 16373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:56.348238 16347 server_base.cc:1034] running on GCE node
W20260926 14:47:56.348172 16375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:56.348400 16372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:56.348790 16347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:56.349465 16347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:56.350697 16347 hybrid_clock.cc:648] HybridClock initialized: now 1790434076350631 us; error 41 us; skew 500 ppm
Sep 26 14:47:56 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:56.353621 16347 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:47:56.354628 16347 webserver.cc:460] Webserver started at http://127.0.98.1:33603/ using document root <none> and password file <none>
I20260926 14:47:56.355245 16347 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:56.355327 16347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:56.359364 16347 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:47:56.361552 16382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:56.362893 16347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:47:56.362996 16347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "0bfd944b41d952bc0ecd0000648ecc50"
server_key_iv: "62057d55f6acdbbbc2b5c0ba393b255a"
server_key_version: "encryptionkey@0"
I20260926 14:47:56.363556 16347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:56.376207 16347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:56.379472 16347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:56.379648 16347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:56.380275 16347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:56.381431 16389 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:47:56.385721 16347 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:47:56.385790 16347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 14:47:56.385842 16347 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:47:56.387717 16347 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:47:56.387784 16347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260926 14:47:56.388065 16389 tablet_bootstrap.cc:492] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Bootstrap starting.
I20260926 14:47:56.398139 16347 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43843
I20260926 14:47:56.398169 16496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43843 every 8 connection(s)
I20260926 14:47:56.399451 16347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:47:56.402076 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 16347
I20260926 14:47:56.402567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40841
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35535
--webserver_port=34883
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:47:56.409399 16389 log.cc:824] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:47:56 dist-test-slave-s7k6 krb5kdc[15727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:56.417481 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.401540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35573 (local address 127.0.98.62:35535)
0926 14:47:56.402915 (+ 1375us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.402934 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.402984 (+ 50us) server_negotiation.cc:407] Connection header received
0926 14:47:56.403883 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.403896 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.404213 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.404527 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:56.405727 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.406846 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.408121 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.408593 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.408754 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.411531 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:56.411538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:56.411554 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:56.411589 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:56.413729 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:56.414404 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:56.414410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:56.414415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:56.414578 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:56.414994 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:56.414998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:56.415000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:56.415516 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:56.415665 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:56.416331 (+ 666us) server_negotiation.cc:299] Negotiation successful
0926 14:47:56.416652 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1011,"thread_start_us":125,"threads_started":1}
I20260926 14:47:56.417503 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.401516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:35573)
0926 14:47:56.402158 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:56.402222 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:56.403393 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:56.404616 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:56.404625 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:56.405031 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:56.405547 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.405562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.407001 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.407003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.407995 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.408006 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.408715 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.408719 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.408901 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.409442 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:56.409459 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:56.411295 (+ 1836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:56.413871 (+ 2576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:56.413877 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:56.413893 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:56.414269 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:56.414681 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:56.414684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:56.414687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:56.414888 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:56.415651 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:56.415658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:56.415969 (+ 311us) client_negotiation.cc:769] Sending connection context
0926 14:47:56.416156 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 14:47:56.416503 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":112,"threads_started":1}
I20260926 14:47:56.418051 16389 tablet_bootstrap.cc:492] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:47:56.418444 16389 tablet_bootstrap.cc:492] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Bootstrap complete.
I20260926 14:47:56.419549 16389 ts_tablet_manager.cc:1397] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.007s
I20260926 14:47:56.420567 16497 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35535
I20260926 14:47:56.420807 16497 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:56.421295 16497 heartbeater.cc:511] Master 127.0.98.62:35535 requested a full tablet report, sending...
I20260926 14:47:56.423962 16309 ts_manager.cc:194] Registered new tserver with Master: 9ae3d5b99ad84ef3b7d53e1062932d6f (127.0.98.1:43843)
I20260926 14:47:56.424341 16389 raft_consensus.cc:348] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:56.424891 16389 raft_consensus.cc:729] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae3d5b99ad84ef3b7d53e1062932d6f, State: Initialized, Role: FOLLOWER
I20260926 14:47:56.425406 16389 consensus_queue.cc:260] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:56.426486 16389 ts_tablet_manager.cc:1428] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260926 14:47:56.428020 16309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35573
I20260926 14:47:56.430264 16497 heartbeater.cc:503] Master 127.0.98.62:35535 was elected leader, sending a full tablet report...
I20260926 14:47:56.435288 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.427503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.0.98.1:43843)
0926 14:47:56.427859 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.427869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.430786 (+ 2917us) server_negotiation.cc:407] Connection header received
0926 14:47:56.431015 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.431021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.431219 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.431386 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:56.431667 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.432499 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.434076 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.434601 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.434733 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.434896 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:56.435094 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 14:47:56.435179 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":92,"threads_started":1}
I20260926 14:47:56.435333 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.427427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43843 (local address 127.0.0.1:56098)
0926 14:47:56.430640 (+ 3213us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:56.430688 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:56.430859 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:56.431388 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:56.431394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:56.431420 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:56.431551 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.431557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.432614 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.432618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.433938 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.433950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.434790 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.434794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.434909 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.434950 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 14:47:56.435058 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 14:47:56.435189 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3038,"thread_start_us":121,"threads_started":1}
I20260926 14:47:56.436359 16431 tablet_service.cc:1514] Processing DeleteTablet for tablet caa4b96740c9418490cf96dd52378807 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56098
I20260926 14:47:56.437206 16506 tablet_replica.cc:330] stopping tablet replica
I20260926 14:47:56.437522 16506 raft_consensus.cc:2229] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:47:56.437790 16506 raft_consensus.cc:2258] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:56.439055 16506 ts_tablet_manager.cc:1905] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:56.443658 16506 ts_tablet_manager.cc:1918] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:47:56.443760 16506 log.cc:1196] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal/wals/caa4b96740c9418490cf96dd52378807
I20260926 14:47:56.444026 16506 ts_tablet_manager.cc:1939] T caa4b96740c9418490cf96dd52378807 P 9ae3d5b99ad84ef3b7d53e1062932d6f: Deleting consensus metadata
I20260926 14:47:56.446058 16296 catalog_manager.cc:4841] TS 9ae3d5b99ad84ef3b7d53e1062932d6f (127.0.98.1:43843): tablet caa4b96740c9418490cf96dd52378807 (table test-table [id=e998b74fdd7f4a0bb3d64725b0498130]) successfully deleted
W20260926 14:47:56.522717 16501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:56.523017 16501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:56.523061 16501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:56.527164 16501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:56.527236 16501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:56.527297 16501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:47:56.532691 16501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34883
--tserver_master_addrs=127.0.98.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:56.533774 16501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:56.534835 16501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:56.544720 16511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:56.544734 16514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:56.544958 16512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:56.544945 16501 server_base.cc:1034] running on GCE node
I20260926 14:47:56.545413 16501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:56.546052 16501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:56.547238 16501 hybrid_clock.cc:648] HybridClock initialized: now 1790434076547200 us; error 48 us; skew 500 ppm
Sep 26 14:47:56 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:56.550033 16501 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:47:56.550966 16501 webserver.cc:460] Webserver started at http://127.0.98.2:34883/ using document root <none> and password file <none>
I20260926 14:47:56.551506 16501 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:56.551581 16501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:56.555539 16501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:47:56.557932 16521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:56.559405 16501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:47:56.559522 16501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "7361b1493e19456896c3d5648556d40f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "e13f55c4d32414ef0e073c44d257f614"
server_key_iv: "790e1e314bdacb9bfa738dd045cdff18"
server_key_version: "encryptionkey@0"
I20260926 14:47:56.560134 16501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:56.576221 16501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:56.579558 16501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:56.579766 16501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:56.580379 16501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:56.581482 16528 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 14:47:56.585422 16528 ts_tablet_manager.cc:1852] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 14:47:56.585844 16528 ts_tablet_manager.cc:1905] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:56.589934 16528 ts_tablet_manager.cc:1918] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:47:56.590037 16528 log.cc:1196] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/caa4b96740c9418490cf96dd52378807
I20260926 14:47:56.590351 16528 ts_tablet_manager.cc:1939] T caa4b96740c9418490cf96dd52378807 P 7361b1493e19456896c3d5648556d40f: Deleting consensus metadata
I20260926 14:47:56.590821 16501 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 14:47:56.590871 16501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260926 14:47:56.590929 16501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:56.590955 16501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:56.599488 16501 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40841
I20260926 14:47:56.599540 16635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40841 every 8 connection(s)
I20260926 14:47:56.600392 16501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:47:56.600991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 16501
I20260926 14:47:56.601377 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41949
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35535
--webserver_port=44331
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 14:47:56 dist-test-slave-s7k6 krb5kdc[15727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:56.614746 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.602357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37621 (local address 127.0.98.62:35535)
0926 14:47:56.602497 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.602501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.603597 (+ 1096us) server_negotiation.cc:407] Connection header received
0926 14:47:56.604636 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.604640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.604706 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.604806 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:56.606143 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.607034 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.608237 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.608599 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.608714 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.611218 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:56.611222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:56.611226 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:56.611263 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:56.612787 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:56.613270 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:56.613273 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:56.613275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:56.613365 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:56.613643 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:56.613646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:56.613647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:56.613871 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:56.613958 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:56.614429 (+ 471us) server_negotiation.cc:299] Negotiation successful
0926 14:47:56.614570 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:47:56.615631 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.602873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:37621)
0926 14:47:56.603403 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:56.603462 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:56.604389 (+ 927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:56.605003 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:56.605011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:56.605365 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:56.605970 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.605985 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.607170 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.607173 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.608107 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.608117 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.608779 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.608782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.608933 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.609523 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:56.609540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:56.611018 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:56.612910 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:56.612915 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:56.612927 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:56.613181 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:56.613442 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:56.613444 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:56.613445 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:56.613568 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:56.613963 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:56.613968 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:56.614217 (+ 249us) client_negotiation.cc:769] Sending connection context
0926 14:47:56.614375 (+ 158us) client_negotiation.cc:240] Negotiation successful
0926 14:47:56.614639 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260926 14:47:56.616920 16636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35535
I20260926 14:47:56.617159 16636 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:56.617676 16636 heartbeater.cc:511] Master 127.0.98.62:35535 requested a full tablet report, sending...
I20260926 14:47:56.618780 16309 ts_manager.cc:194] Registered new tserver with Master: 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841)
I20260926 14:47:56.619571 16309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37621
W20260926 14:47:56.717221 16638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:47:56.717497 16638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:47:56.717541 16638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:47:56.721715 16638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:47:56.721791 16638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:47:56.721851 16638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:47:56.727404 16638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44331
--tserver_master_addrs=127.0.98.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:47:56.728457 16638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:47:56.729584 16638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:47:56.739398 16644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:56.739491 16638 server_base.cc:1034] running on GCE node
W20260926 14:47:56.739408 16645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:47:56.739391 16647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:47:56.740123 16638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:47:56.740979 16638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:47:56.742172 16638 hybrid_clock.cc:648] HybridClock initialized: now 1790434076742153 us; error 37 us; skew 500 ppm
Sep 26 14:47:56 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:47:56.744943 16638 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:47:56.745934 16638 webserver.cc:460] Webserver started at http://127.0.98.3:44331/ using document root <none> and password file <none>
I20260926 14:47:56.746548 16638 fs_manager.cc:359] Metadata directory not provided
I20260926 14:47:56.746634 16638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:47:56.750945 16638 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 14:47:56.753224 16654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:56.754529 16638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:47:56.754652 16638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "0f772a2f91c34cd494c6dae3a9faf35f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "27ce6ba81de7043f645ff16adf8c501b"
server_key_iv: "c2bc914ca48781226e2af2babe5ba028"
server_key_version: "encryptionkey@0"
I20260926 14:47:56.755245 16638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:47:56.765789 16638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:47:56.769234 16638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:47:56.769410 16638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:47:56.769981 16638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:47:56.770828 16638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:47:56.770879 16638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:56.770924 16638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:47:56.770951 16638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:47:56.779414 16638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41949
I20260926 14:47:56.779470 16767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41949 every 8 connection(s)
I20260926 14:47:56.780452 16638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:47:56.788349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 16638
Sep 26 14:47:56 dist-test-slave-s7k6 krb5kdc[15727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:47:56.794862 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.782407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52941 (local address 127.0.98.62:35535)
0926 14:47:56.782568 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.782571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.783740 (+ 1169us) server_negotiation.cc:407] Connection header received
0926 14:47:56.784629 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.784633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.784702 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.784774 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:56.785984 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.786507 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.787797 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.788019 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.788111 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.790909 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:56.790913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:56.790916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:56.790943 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:56.792514 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:56.793178 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:56.793181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:56.793182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:56.793275 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:56.793658 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:56.793660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:56.793661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:56.793835 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:56.793909 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:56.794473 (+ 564us) server_negotiation.cc:299] Negotiation successful
0926 14:47:56.794665 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 14:47:56.795682 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.782848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:52941)
0926 14:47:56.783517 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:56.783562 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:56.784418 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:56.784965 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:56.784972 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:56.785339 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:47:56.785833 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.785847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.786625 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.786628 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.787667 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.787679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.788167 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.788180 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.788350 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.788994 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:47:56.789013 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:47:56.790705 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:47:56.792688 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:56.792695 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:56.792712 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:56.793055 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:56.793364 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:47:56.793368 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:47:56.793370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:47:56.793561 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:47:56.793934 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:47:56.793942 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:47:56.794209 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 14:47:56.794376 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 14:47:56.794751 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":107,"threads_started":1}
I20260926 14:47:56.796912 16768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35535
I20260926 14:47:56.797188 16768 heartbeater.cc:464] Registering TS with master...
I20260926 14:47:56.797729 16768 heartbeater.cc:511] Master 127.0.98.62:35535 requested a full tablet report, sending...
I20260926 14:47:56.798544 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.789051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49776 (local address 127.0.98.62:35535)
0926 14:47:56.789356 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.789360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.789433 (+ 73us) server_negotiation.cc:407] Connection header received
0926 14:47:56.789558 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.789560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.789597 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.789669 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:56.790417 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.791079 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.792165 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.792519 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.792624 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.793976 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:56.793978 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:56.793980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:56.793999 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:56.795512 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:56.795953 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:56.795957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:56.795958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:56.796032 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:56.796386 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:56.796389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:56.796391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:56.796589 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:56.796784 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:56.798274 (+ 1490us) server_negotiation.cc:299] Negotiation successful
0926 14:47:56.798388 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":93,"threads_started":1}
I20260926 14:47:56.798904 16309 ts_manager.cc:194] Registered new tserver with Master: 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949)
I20260926 14:47:56.799508 16309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52941
I20260926 14:47:56.800431 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:47:56.804505 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.801087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.0.98.62:35535)
0926 14:47:56.801199 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.801204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.801482 (+ 278us) server_negotiation.cc:407] Connection header received
0926 14:47:56.801629 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.801631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.801771 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.801852 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:47:56.802120 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.802650 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.803512 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.803770 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.803846 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.804081 (+ 235us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:47:56.804205 (+ 124us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 14:47:56.804401 (+ 196us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49788: token expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 14:47:56.804572 16772 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49788: token expired
W20260926 14:47:56.805585 16235 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:35535)
I20260926 14:47:56.813648 16772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.805956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.0.98.62:35535)
0926 14:47:56.806076 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.806080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.806118 (+ 38us) server_negotiation.cc:407] Connection header received
0926 14:47:56.806175 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.806178 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.806231 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.806299 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:47:56.807097 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.807871 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.808994 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.809225 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.809311 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.810834 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:47:56.810838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:47:56.810841 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:47:56.810869 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:47:56.812057 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:56.812460 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:56.812463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:56.812465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:56.812543 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:47:56.812789 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:47:56.812793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:47:56.812794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:47:56.813017 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:47:56.813136 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:56.813374 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 14:47:56.813494 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:47:56.816339 16308 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:47:56.819885 16308 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:47:56.828177 16431 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcf57495fdf4ec589b888b7d49f53cd (DEFAULT_TABLE table=test-table [id=1505bd2f7a9f48d0ad050f8e06a3c9e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:56.828904 16431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcf57495fdf4ec589b888b7d49f53cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:56.832005 16389 tablet_bootstrap.cc:492] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: Bootstrap starting.
I20260926 14:47:56.832320 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.827075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41949 (local address 127.0.0.1:59708)
0926 14:47:56.827260 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:56.827275 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:56.827357 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:56.828606 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:56.828610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:56.828629 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:56.828748 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.828754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.829931 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.829934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.830733 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.830743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.831913 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.831917 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.832016 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.832043 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 14:47:56.832108 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:47:56.832190 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 14:47:56.832818 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.827550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59708 (local address 127.0.98.3:41949)
0926 14:47:56.828074 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.828087 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.828111 (+ 24us) server_negotiation.cc:407] Connection header received
0926 14:47:56.828197 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.828203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.828422 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.828634 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:56.828848 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.829818 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.830917 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.831766 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.831888 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.832047 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:56.832214 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 14:47:56.832322 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":301,"threads_started":1}
I20260926 14:47:56.832950 16389 tablet_bootstrap.cc:654] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:56.833493 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.827298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40841 (local address 127.0.0.1:56144)
0926 14:47:56.827561 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:56.827574 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:56.827664 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:56.829079 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:56.829083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:56.829101 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:56.829220 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.829227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.830828 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.830831 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.831968 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.831978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.833097 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.833102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.833204 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.833229 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:47:56.833288 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 14:47:56.833357 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":120,"threads_started":1}
I20260926 14:47:56.834124 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.827865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.0.98.2:40841)
0926 14:47:56.828551 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.828564 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.828597 (+ 33us) server_negotiation.cc:407] Connection header received
0926 14:47:56.828693 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.828698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.828915 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.829383 (+ 468us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:56.829561 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.830681 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.832106 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.832959 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.833682 (+ 723us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.833836 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:56.833890 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 14:47:56.833978 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":140,"threads_started":1}
I20260926 14:47:56.834823 16389 tablet_bootstrap.cc:492] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: No bootstrap required, opened a new log
I20260926 14:47:56.834918 16389 ts_tablet_manager.cc:1397] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 14:47:56.835597 16389 raft_consensus.cc:348] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:56.835731 16389 raft_consensus.cc:374] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:56.835770 16389 raft_consensus.cc:729] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae3d5b99ad84ef3b7d53e1062932d6f, State: Initialized, Role: FOLLOWER
I20260926 14:47:56.835897 16389 consensus_queue.cc:260] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:56.836167 16389 ts_tablet_manager.cc:1428] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 14:47:56.836078 16570 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcf57495fdf4ec589b888b7d49f53cd (DEFAULT_TABLE table=test-table [id=1505bd2f7a9f48d0ad050f8e06a3c9e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:56.836614 16702 tablet_service.cc:1467] Processing CreateTablet for tablet 3dcf57495fdf4ec589b888b7d49f53cd (DEFAULT_TABLE table=test-table [id=1505bd2f7a9f48d0ad050f8e06a3c9e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:47:56.837278 16570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcf57495fdf4ec589b888b7d49f53cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:56.837623 16702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dcf57495fdf4ec589b888b7d49f53cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:47:56.842187 16528 tablet_bootstrap.cc:492] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f: Bootstrap starting.
I20260926 14:47:56.844626 16528 tablet_bootstrap.cc:654] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:56.845366 16781 tablet_bootstrap.cc:492] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f: Bootstrap starting.
I20260926 14:47:56.845535 16528 log.cc:824] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:56.847695 16528 tablet_bootstrap.cc:492] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f: No bootstrap required, opened a new log
I20260926 14:47:56.847878 16528 ts_tablet_manager.cc:1397] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 14:47:56.848140 16781 tablet_bootstrap.cc:654] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f: Neither blocks nor log segments found. Creating new log.
I20260926 14:47:56.849130 16781 log.cc:824] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:47:56.850891 16781 tablet_bootstrap.cc:492] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f: No bootstrap required, opened a new log
I20260926 14:47:56.850793 16528 raft_consensus.cc:348] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:56.851035 16528 raft_consensus.cc:374] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:56.851079 16781 ts_tablet_manager.cc:1397] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:47:56.851096 16528 raft_consensus.cc:729] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7361b1493e19456896c3d5648556d40f, State: Initialized, Role: FOLLOWER
I20260926 14:47:56.851534 16528 consensus_queue.cc:260] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:56.852159 16636 heartbeater.cc:503] Master 127.0.98.62:35535 was elected leader, sending a full tablet report...
I20260926 14:47:56.852474 16528 ts_tablet_manager.cc:1428] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:47:56.854784 16781 raft_consensus.cc:348] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:56.855072 16781 raft_consensus.cc:374] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:47:56.855137 16781 raft_consensus.cc:729] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f772a2f91c34cd494c6dae3a9faf35f, State: Initialized, Role: FOLLOWER
I20260926 14:47:56.855552 16781 consensus_queue.cc:260] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:56.856199 16768 heartbeater.cc:503] Master 127.0.98.62:35535 was elected leader, sending a full tablet report...
I20260926 14:47:56.856485 16781 ts_tablet_manager.cc:1428] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260926 14:47:56.900785 16498 tablet.cc:2369] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:47:56.930145 16785 raft_consensus.cc:482] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:47:56.930349 16785 raft_consensus.cc:504] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:56.931556 16785 leader_election.cc:288] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841), 9ae3d5b99ad84ef3b7d53e1062932d6f (127.0.98.1:43843)
I20260926 14:47:56.935272 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.931826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40841 (local address 127.0.98.3:34317)
0926 14:47:56.932006 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:56.932022 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:56.932130 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:56.932421 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:56.932424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:56.932443 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:56.932553 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.932558 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.933273 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.933276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.934128 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.934138 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.934831 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.934835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.934939 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.934979 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 14:47:56.935044 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:47:56.935125 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":3}
I20260926 14:47:56.935272 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.932070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34317 (local address 127.0.98.2:40841)
0926 14:47:56.932203 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.932206 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.932217 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:47:56.932275 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.932277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.932319 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.932391 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:56.932664 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.933164 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.934270 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.934685 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.934809 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.934878 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:56.935070 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 14:47:56.935143 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:47:56.935396 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.931975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51049 (local address 127.0.98.1:43843)
0926 14:47:56.932103 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.932108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.932386 (+ 278us) server_negotiation.cc:407] Connection header received
0926 14:47:56.932470 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.932474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.932520 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.932580 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:47:56.932850 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.933344 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.934370 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.934834 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.934931 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.934996 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:56.935197 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 14:47:56.935278 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:47:56.936024 16590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcf57495fdf4ec589b888b7d49f53cd" candidate_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7361b1493e19456896c3d5648556d40f" is_pre_election: true
I20260926 14:47:56.936101 16786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.931869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43843 (local address 127.0.98.3:51049)
0926 14:47:56.932289 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:56.932299 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:47:56.932387 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:47:56.932605 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:47:56.932608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:47:56.932631 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:47:56.932745 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.932750 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.933452 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.933456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.934242 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:47:56.934251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.934946 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.934950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:47:56.935085 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.935102 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:47:56.935169 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:47:56.935242 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":21,"thread_start_us":121,"threads_started":1}
I20260926 14:47:56.936384 16590 raft_consensus.cc:2454] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f772a2f91c34cd494c6dae3a9faf35f in term 0.
I20260926 14:47:56.936738 16451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcf57495fdf4ec589b888b7d49f53cd" candidate_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" is_pre_election: true
I20260926 14:47:56.936934 16655 leader_election.cc:302] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f772a2f91c34cd494c6dae3a9faf35f, 7361b1493e19456896c3d5648556d40f; no voters:
I20260926 14:47:56.937016 16451 raft_consensus.cc:2454] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f772a2f91c34cd494c6dae3a9faf35f in term 0.
I20260926 14:47:56.937255 16785 raft_consensus.cc:2781] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:47:56.937348 16785 raft_consensus.cc:482] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:47:56.937402 16785 raft_consensus.cc:3037] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:56.938457 16785 raft_consensus.cc:504] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:56.938844 16785 leader_election.cc:288] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [CANDIDATE]: Term 1 election: Requested vote from peers 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841), 9ae3d5b99ad84ef3b7d53e1062932d6f (127.0.98.1:43843)
I20260926 14:47:56.939147 16590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcf57495fdf4ec589b888b7d49f53cd" candidate_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7361b1493e19456896c3d5648556d40f"
I20260926 14:47:56.939203 16451 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3dcf57495fdf4ec589b888b7d49f53cd" candidate_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f"
I20260926 14:47:56.939267 16590 raft_consensus.cc:3037] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:56.939344 16451 raft_consensus.cc:3037] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:47:56.940330 16590 raft_consensus.cc:2454] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f772a2f91c34cd494c6dae3a9faf35f in term 1.
I20260926 14:47:56.940456 16451 raft_consensus.cc:2454] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f772a2f91c34cd494c6dae3a9faf35f in term 1.
I20260926 14:47:56.940689 16655 leader_election.cc:302] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f772a2f91c34cd494c6dae3a9faf35f, 7361b1493e19456896c3d5648556d40f; no voters:
I20260926 14:47:56.940846 16785 raft_consensus.cc:2781] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:47:56.941061 16785 raft_consensus.cc:686] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 1 LEADER]: Becoming Leader. State: Replica: 0f772a2f91c34cd494c6dae3a9faf35f, State: Running, Role: LEADER
I20260926 14:47:56.941421 16785 consensus_queue.cc:237] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:47:56.944582 16308 catalog_manager.cc:5495] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } health_report { overall_health: UNKNOWN } } }
I20260926 14:47:56.959008 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.955590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59718 (local address 127.0.98.3:41949)
0926 14:47:56.955786 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 14:47:56.955790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:47:56.955804 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:47:56.955876 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:47:56.955878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:47:56.955931 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:47:56.956018 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:47:56.956241 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.956743 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.957562 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:47:56.957782 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:47:56.957868 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:47:56.958149 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:47:56.958577 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:47:56.958680 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:47:56.958793 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 14:47:56.958852 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 14:47:56.964986 16590 raft_consensus.cc:1261] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f [term 1 FOLLOWER]: Refusing update from remote peer 0f772a2f91c34cd494c6dae3a9faf35f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:56.964986 16451 raft_consensus.cc:1261] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Refusing update from remote peer 0f772a2f91c34cd494c6dae3a9faf35f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:47:56.965618 16785 consensus_queue.cc:1035] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:47:56.965811 16787 consensus_queue.cc:1035] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:47:56.981640 16308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:47:56.981824 16308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:47:56.984309 16308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Sending DeleteTablet for 3 replicas of tablet 3dcf57495fdf4ec589b888b7d49f53cd
I20260926 14:47:56.984988 16431 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dcf57495fdf4ec589b888b7d49f53cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56098
I20260926 14:47:56.985077 16702 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dcf57495fdf4ec589b888b7d49f53cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59708
I20260926 14:47:56.985116 16570 tablet_service.cc:1514] Processing DeleteTablet for tablet 3dcf57495fdf4ec589b888b7d49f53cd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:47:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56144
I20260926 14:47:56.985409 16799 tablet_replica.cc:330] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: stopping tablet replica
I20260926 14:47:56.985538 16799 raft_consensus.cc:2229] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:47:56.985716 16799 raft_consensus.cc:2258] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:56.985950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 16347
I20260926 14:47:56.986042 16801 tablet_replica.cc:330] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f: stopping tablet replica
I20260926 14:47:56.986369 16801 raft_consensus.cc:2229] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:47:56.986716 16801 raft_consensus.cc:2258] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:56.988394 16801 ts_tablet_manager.cc:1905] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:56.990617 16801 ts_tablet_manager.cc:1918] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:47:56.990701 16801 log.cc:1196] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/3dcf57495fdf4ec589b888b7d49f53cd
I20260926 14:47:56.990942 16801 ts_tablet_manager.cc:1939] T 3dcf57495fdf4ec589b888b7d49f53cd P 7361b1493e19456896c3d5648556d40f: Deleting consensus metadata
I20260926 14:47:56.992231 16293 catalog_manager.cc:4841] TS 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841): tablet 3dcf57495fdf4ec589b888b7d49f53cd (table test-table [id=1505bd2f7a9f48d0ad050f8e06a3c9e6]) successfully deleted
W20260926 14:47:56.993635 16296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43843 (error 108)
I20260926 14:47:56.994210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 16501
I20260926 14:47:56.995072 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:47:56.994364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43843 (local address 127.0.0.1:56114)
0926 14:47:56.994892 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 14:47:56.994974 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434}
W20260926 14:47:56.995674 16296 catalog_manager.cc:4584] TS 9ae3d5b99ad84ef3b7d53e1062932d6f (127.0.98.1:43843): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3dcf57495fdf4ec589b888b7d49f53cd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43843: connect: Connection refused (error 111)
I20260926 14:47:56.996349 16800 tablet_replica.cc:330] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f: stopping tablet replica
I20260926 14:47:56.996651 16800 raft_consensus.cc:2229] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:47:56.997010 16800 raft_consensus.cc:2258] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:47:56.998360 16800 ts_tablet_manager.cc:1905] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:47:57.000581 16800 ts_tablet_manager.cc:1918] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:47:57.000677 16800 log.cc:1196] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/3dcf57495fdf4ec589b888b7d49f53cd
I20260926 14:47:57.001016 16800 ts_tablet_manager.cc:1939] T 3dcf57495fdf4ec589b888b7d49f53cd P 0f772a2f91c34cd494c6dae3a9faf35f: Deleting consensus metadata
I20260926 14:47:57.001163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 16638
I20260926 14:47:57.006528 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 16276
I20260926 14:48:17.012579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35535
--webserver_interface=127.0.98.62
--webserver_port=35509
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:17.125933 16803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:17.126231 16803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:17.126272 16803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:17.130561 16803 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 14:48:17.130626 16803 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 14:48:17.130641 16803 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 14:48:17.130656 16803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:17.130672 16803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:17.130688 16803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:17.130707 16803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:17.130736 16803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:17.136612 16803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:17.137728 16803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:17.138815 16803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:17.148082 16808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:17.148279 16803 server_base.cc:1034] running on GCE node
W20260926 14:48:17.148082 16811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:17.148162 16809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:17.149080 16803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:17.150298 16803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:17.151476 16803 hybrid_clock.cc:648] HybridClock initialized: now 1790434097151456 us; error 37 us; skew 500 ppm
Sep 26 14:48:17 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:17.154357 16803 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:48:17.155470 16803 webserver.cc:460] Webserver started at http://127.0.98.62:35509/ using document root <none> and password file <none>
I20260926 14:48:17.156049 16803 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:17.156122 16803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:17.160254 16803 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:48:17.162599 16818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:17.163975 16803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:17.164106 16803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "9dfcbc0210034697b346ed5d0d11749c"
format_stamp: "Formatted at 2026-09-26 14:47:34 on dist-test-slave-s7k6"
server_key: "e43142b99cc4c3f7bb61e5bd824e878b"
server_key_iv: "436815f1f0e2e50640d4af1e10061534"
server_key_version: "encryptionkey@0"
I20260926 14:48:17.164700 16803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:17.198141 16803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:17.201232 16803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:17.201435 16803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:17.208721 16803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35535
I20260926 14:48:17.209007 16870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35535 every 8 connection(s)
I20260926 14:48:17.209860 16803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:17.214401 16871 sys_catalog.cc:263] Verifying existing consensus state
I20260926 14:48:17.216326 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Bootstrap starting.
I20260926 14:48:17.219197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 16803
I20260926 14:48:17.219655 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43843
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35535
--webserver_port=33603
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:48:17.234516 16871 log.cc:824] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:17.246470 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:48:17.246896 16871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Bootstrap complete.
I20260926 14:48:17.250964 16871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:48:17.251389 16871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfcbc0210034697b346ed5d0d11749c, State: Initialized, Role: FOLLOWER
I20260926 14:48:17.251920 16871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:48:17.252051 16871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:17.252116 16871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:17.252213 16871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 2 FOLLOWER]: Advancing to term 3
I20260926 14:48:17.254264 16871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:48:17.254608 16871 leader_election.cc:302] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dfcbc0210034697b346ed5d0d11749c; no voters:
I20260926 14:48:17.254944 16871 leader_election.cc:288] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 14:48:17.255054 16875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 3 FOLLOWER]: Leader election won for term 3
I20260926 14:48:17.255244 16875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [term 3 LEADER]: Becoming Leader. State: Replica: 9dfcbc0210034697b346ed5d0d11749c, State: Running, Role: LEADER
I20260926 14:48:17.255609 16875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } }
I20260926 14:48:17.256232 16871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:17.257109 16877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dfcbc0210034697b346ed5d0d11749c. Latest consensus state: current_term: 3 leader_uuid: "9dfcbc0210034697b346ed5d0d11749c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } } }
I20260926 14:48:17.257249 16877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:17.257500 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9dfcbc0210034697b346ed5d0d11749c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfcbc0210034697b346ed5d0d11749c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35535 } } }
I20260926 14:48:17.257578 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:17.257612 16884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:17.262871 16884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:17.263592 16884 catalog_manager.cc:1259] Loaded cluster ID: a75a1021a5804640b182b744a6832c8a
I20260926 14:48:17.263653 16884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:17.265631 16884 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:17.266287 16884 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Loaded TSK: 0
I20260926 14:48:17.266360 16884 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9dfcbc0210034697b346ed5d0d11749c: Loaded TSK: 1
F20260926 14:48:17.267045 16884 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fd01b6cf9ad google::LogMessage::Fail() at ??:0
@ 0x7fd01b6d288a google::LogMessage::SendToLog() at ??:0
@ 0x7fd01b6cf4de google::LogMessage::Flush() at ??:0
@ 0x7fd01b6d0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd01e39318d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fd01e38bf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fd0260347bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fd02600e99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fd02600f178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7fd026050df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fd02606f82b std::function<>::operator()() at ??:0
@ 0x7fd02600eb4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fd02600ff3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fd02600c579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fd026050640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd01d9213d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd01d921c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd01d9233f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fd01d912512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd01ef566db start_thread at ??:0
@ 0x7fd01a1cd61f clone at ??:0
W20260926 14:48:17.369871 16873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:17.370231 16873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:17.370299 16873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:17.374925 16873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:17.374998 16873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:17.375094 16873 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:48:17.382469 16873 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33603
--tserver_master_addrs=127.0.98.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:17.383793 16873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:17.385025 16873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:17.396519 16918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:17.396531 16919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:17.396552 16921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:17.397141 16873 server_base.cc:1034] running on GCE node
I20260926 14:48:17.397579 16873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:17.398262 16873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:17.399453 16873 hybrid_clock.cc:648] HybridClock initialized: now 1790434097399423 us; error 43 us; skew 500 ppm
Sep 26 14:48:17 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:17.402294 16873 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:48:17.403376 16873 webserver.cc:460] Webserver started at http://127.0.98.1:33603/ using document root <none> and password file <none>
I20260926 14:48:17.403957 16873 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:17.404031 16873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:17.408278 16873 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:48:17.410593 16928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:17.411952 16873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:17.412076 16873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "0bfd944b41d952bc0ecd0000648ecc50"
server_key_iv: "62057d55f6acdbbbc2b5c0ba393b255a"
server_key_version: "encryptionkey@0"
I20260926 14:48:17.412652 16873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:17.449544 16873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:17.452986 16873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:17.453182 16873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:17.453811 16873 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:17.455047 16935 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 14:48:17.460793 16873 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 14:48:17.460876 16873 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 14:48:17.460945 16873 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 14:48:17.462822 16873 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 14:48:17.462880 16873 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260926 14:48:17.463212 16935 tablet_bootstrap.cc:492] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: Bootstrap starting.
I20260926 14:48:17.472782 16873 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43843
I20260926 14:48:17.472949 17042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43843 every 8 connection(s)
I20260926 14:48:17.473889 16873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:48:17.474517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 16873
I20260926 14:48:17.475008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40841
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35535
--webserver_port=34883
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:48:17.488026 16935 log.cc:824] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:17.497664 16935 tablet_bootstrap.cc:492] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 14:48:17.498124 16935 tablet_bootstrap.cc:492] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: Bootstrap complete.
I20260926 14:48:17.499538 16935 ts_tablet_manager.cc:1397] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: Time spent bootstrapping tablet: real 0.037s user 0.025s sys 0.008s
I20260926 14:48:17.502889 16935 raft_consensus.cc:348] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:48:17.503453 16935 raft_consensus.cc:729] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae3d5b99ad84ef3b7d53e1062932d6f, State: Initialized, Role: FOLLOWER
I20260926 14:48:17.504019 16935 consensus_queue.cc:260] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:48:17.505131 16935 ts_tablet_manager.cc:1428] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260926 14:48:17.519694 17046 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:48:17.476971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:48635)
0926 14:48:17.477560 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:17.477617 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:17.478663 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:17.518468 (+ 39805us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":155,"threads_started":1}
W20260926 14:48:17.520242 17043 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35535 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35535: Client connection negotiation failed: client connection to 127.0.98.62:35535: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 14:48:17.521054 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:17.520719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:49419)
0926 14:48:17.520865 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:17.520953 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 14:48:17.521883 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:17.521631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:52591)
0926 14:48:17.521741 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:17.521819 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 14:48:17.522066 17043 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 14:48:17.590859 17045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:17.591138 17045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:17.591194 17045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:17.595443 17045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:17.595507 17045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:17.595582 17045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:48:17.601120 17045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40841
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=34883
--tserver_master_addrs=127.0.98.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:17.602188 17045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:17.603255 17045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:17.613166 17056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:17.613173 17054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:17.613168 17053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:17.613466 17045 server_base.cc:1034] running on GCE node
I20260926 14:48:17.613943 17045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:17.614622 17045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:17.615792 17045 hybrid_clock.cc:648] HybridClock initialized: now 1790434097615764 us; error 45 us; skew 500 ppm
Sep 26 14:48:17 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:17.618726 17045 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:48:17.619712 17045 webserver.cc:460] Webserver started at http://127.0.98.2:34883/ using document root <none> and password file <none>
I20260926 14:48:17.620283 17045 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:17.620357 17045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:17.624497 17045 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:48:17.626888 17063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:17.628327 17045 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:17.628453 17045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "7361b1493e19456896c3d5648556d40f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "e13f55c4d32414ef0e073c44d257f614"
server_key_iv: "790e1e314bdacb9bfa738dd045cdff18"
server_key_version: "encryptionkey@0"
I20260926 14:48:17.629050 17045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:17.648934 17045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:17.651887 17045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:17.652065 17045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:17.652645 17045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:17.653478 17045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:17.653524 17045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:17.653577 17045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:17.653602 17045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:17.662034 17045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40841
I20260926 14:48:17.662053 17176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40841 every 8 connection(s)
I20260926 14:48:17.663152 17045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:48:17.667034 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:17.665419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:58359)
0926 14:48:17.666112 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:17.666229 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":131,"threads_started":1}
W20260926 14:48:17.667527 17177 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35535 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35535: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
I20260926 14:48:17.668125 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:17.667907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:34481)
0926 14:48:17.668010 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:17.668059 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 14:48:17.668725 17179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:17.668528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:55143)
0926 14:48:17.668614 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:17.668677 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 14:48:17.668869 17177 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:48:17.671933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 17045
I20260926 14:48:17.672300 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41949
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35535
--webserver_port=44331
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:17.784786 17180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:17.785059 17180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:17.785102 17180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:17.789238 17180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:17.789306 17180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:17.789366 17180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:48:17.794800 17180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44331
--tserver_master_addrs=127.0.98.62:35535
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:17.795846 17180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:17.796821 17180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:17.806378 17185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:17.806479 17188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:17.806629 17186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:17.806856 17180 server_base.cc:1034] running on GCE node
I20260926 14:48:17.807265 17180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:17.807938 17180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:17.809100 17180 hybrid_clock.cc:648] HybridClock initialized: now 1790434097809077 us; error 39 us; skew 500 ppm
Sep 26 14:48:17 dist-test-slave-s7k6 krb5kdc[15727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:17.811944 17180 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:48:17.812894 17180 webserver.cc:460] Webserver started at http://127.0.98.3:44331/ using document root <none> and password file <none>
I20260926 14:48:17.813459 17180 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:17.813529 17180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:17.817606 17180 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:48:17.819929 17195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:17.821362 17180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:17.821473 17180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "0f772a2f91c34cd494c6dae3a9faf35f"
format_stamp: "Formatted at 2026-09-26 14:47:35 on dist-test-slave-s7k6"
server_key: "27ce6ba81de7043f645ff16adf8c501b"
server_key_iv: "c2bc914ca48781226e2af2babe5ba028"
server_key_version: "encryptionkey@0"
I20260926 14:48:17.822042 17180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:17.839371 17180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:17.842730 17180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:17.842942 17180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:17.843526 17180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:17.844344 17180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:17.844390 17180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:17.844430 17180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:17.844449 17180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:17.853252 17180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41949
I20260926 14:48:17.853281 17308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41949 every 8 connection(s)
I20260926 14:48:17.854310 17180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:48:17.858248 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:17.856617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:56939)
0926 14:48:17.857371 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:17.857491 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":114,"threads_started":1}
I20260926 14:48:17.858479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 17180
W20260926 14:48:17.858728 17309 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35535 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:35535: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
I20260926 14:48:17.859373 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:17.859121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:33293)
0926 14:48:17.859246 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:17.859302 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 14:48:17.859989 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:17.859791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:60091)
0926 14:48:17.859891 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:17.859933 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:48:17.860145 17309 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 14:48:18.523239 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:18.522696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:44891)
0926 14:48:18.523055 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:18.523129 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":162,"threads_started":1}
I20260926 14:48:18.669951 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:18.669408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:58773)
0926 14:48:18.669767 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:18.669831 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":88,"threads_started":1}
I20260926 14:48:18.861332 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:18.860789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:34623)
0926 14:48:18.861085 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:18.861212 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":78,"threads_started":1}
I20260926 14:48:19.226600 17316 raft_consensus.cc:482] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:48:19.226840 17316 raft_consensus.cc:504] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:48:19.228093 17316 leader_election.cc:288] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949), 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841)
Sep 26 14:48:19 dist-test-slave-s7k6 krb5kdc[15727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 14:48:19 dist-test-slave-s7k6 krb5kdc[15727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 14:48:19.242396 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:19.228468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41949 (local address 127.0.98.1:60287)
0926 14:48:19.228720 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:19.228737 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:19.228870 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:19.230527 (+ 1657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:19.230533 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:19.231069 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:19.231634 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:19.231650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:19.233167 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:19.233171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:19.234340 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:19.234352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:19.235157 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:19.235161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:19.235492 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:19.236204 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:48:19.236230 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:19.237870 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:19.240044 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:19.240051 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:19.240066 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:19.240356 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:19.240674 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:19.240677 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:19.240679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:19.240801 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:19.241413 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:19.241420 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:19.241668 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 14:48:19.241833 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 14:48:19.242206 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":319,"spinlock_wait_cycles":17024,"thread_start_us":87,"threads_started":1}
I20260926 14:48:19.242413 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:19.228519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60287 (local address 127.0.98.3:41949)
0926 14:48:19.228847 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:19.228878 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:19.228936 (+ 58us) server_negotiation.cc:407] Connection header received
0926 14:48:19.229751 (+ 815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:19.229759 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:19.230142 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:19.230402 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:19.231811 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:19.233023 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:19.234489 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:19.235033 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:19.235205 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:19.238103 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:19.238109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:19.238121 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:19.238151 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:19.239928 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:19.240466 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:19.240471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:19.240475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:19.240589 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:19.240888 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:19.240891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:19.240893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:19.241304 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:19.241424 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:19.241961 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 14:48:19.242217 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":117,"threads_started":1}
I20260926 14:48:19.242444 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:19.228840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40841 (local address 127.0.98.1:54261)
0926 14:48:19.229089 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:19.229100 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:19.229170 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:19.230675 (+ 1505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:19.230678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:19.231067 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:19.231633 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:19.231650 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:19.233167 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:19.233171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:19.234344 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:19.234387 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:19.235160 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:19.235168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:19.235488 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:19.236236 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:48:19.236248 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:19.238200 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:19.240122 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:19.240125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:19.240127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:19.240381 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:19.240726 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:19.240729 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:19.240731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:19.240808 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:19.241416 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:19.241425 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:19.241668 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 14:48:19.241835 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 14:48:19.242287 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":464,"spinlock_wait_cycles":9984,"thread_start_us":69,"threads_started":1}
I20260926 14:48:19.242416 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:19.229101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54261 (local address 127.0.98.2:40841)
0926 14:48:19.229376 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:19.229387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:19.229417 (+ 30us) server_negotiation.cc:407] Connection header received
0926 14:48:19.230067 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:19.230073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:19.230412 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:19.230712 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:19.231810 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:19.232973 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:19.234503 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:19.235032 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:19.235208 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:19.238384 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:19.238390 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:19.238403 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:19.238438 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:19.240013 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:19.240515 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:19.240520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:19.240524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:19.240637 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:19.240886 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:19.240890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:19.240892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:19.241304 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:19.241425 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:19.241972 (+ 547us) server_negotiation.cc:299] Negotiation successful
0926 14:48:19.242217 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":59,"threads_started":1}
W20260926 14:48:19.245173 16929 leader_election.cc:341] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949): Not found: Tablet not found: 3dcf57495fdf4ec589b888b7d49f53cd
W20260926 14:48:19.245406 16929 leader_election.cc:341] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841): Not found: Tablet not found: 3dcf57495fdf4ec589b888b7d49f53cd
I20260926 14:48:19.245486 16929 leader_election.cc:302] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ae3d5b99ad84ef3b7d53e1062932d6f; no voters: 0f772a2f91c34cd494c6dae3a9faf35f, 7361b1493e19456896c3d5648556d40f
I20260926 14:48:19.245752 17316 raft_consensus.cc:2726] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:48:19.524459 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:19.524149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:54923)
0926 14:48:19.524305 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:19.524371 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 14:48:19.671360 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:19.670883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:34827)
0926 14:48:19.671155 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:19.671232 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":78,"threads_started":1}
I20260926 14:48:19.862848 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:19.862267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:60281)
0926 14:48:19.862592 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:19.862697 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":79,"threads_started":1}
I20260926 14:48:20.525725 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:20.525225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:47479)
0926 14:48:20.525510 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:20.525578 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260926 14:48:20.672624 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:20.672189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:49951)
0926 14:48:20.672459 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:20.672522 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":83,"threads_started":1}
I20260926 14:48:20.864224 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:20.863753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:44729)
0926 14:48:20.864028 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:20.864098 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":79,"threads_started":1}
I20260926 14:48:21.275856 17326 raft_consensus.cc:482] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:48:21.275982 17326 raft_consensus.cc:504] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:48:21.276329 17326 leader_election.cc:288] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949), 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841)
W20260926 14:48:21.277204 16929 leader_election.cc:341] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949): Not found: Tablet not found: 3dcf57495fdf4ec589b888b7d49f53cd
W20260926 14:48:21.277318 16929 leader_election.cc:341] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841): Not found: Tablet not found: 3dcf57495fdf4ec589b888b7d49f53cd
I20260926 14:48:21.277354 16929 leader_election.cc:302] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ae3d5b99ad84ef3b7d53e1062932d6f; no voters: 0f772a2f91c34cd494c6dae3a9faf35f, 7361b1493e19456896c3d5648556d40f
I20260926 14:48:21.277518 17326 raft_consensus.cc:2726] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:48:21.526993 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:21.526493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:41085)
0926 14:48:21.526815 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:21.526890 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":84,"threads_started":1}
I20260926 14:48:21.673960 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:21.673479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:55655)
0926 14:48:21.673771 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:21.673839 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260926 14:48:21.865427 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:21.865015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:35689)
0926 14:48:21.865268 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:21.865329 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":74,"threads_started":1}
I20260926 14:48:22.528388 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:22.527908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:41273)
0926 14:48:22.528217 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:22.528281 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":73,"threads_started":1}
I20260926 14:48:22.675171 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:22.674732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:57805)
0926 14:48:22.675008 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:22.675067 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260926 14:48:22.866824 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:22.866298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:43461)
0926 14:48:22.866618 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:22.866685 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":101,"threads_started":1}
I20260926 14:48:23.021426 17333 raft_consensus.cc:482] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:48:23.021562 17333 raft_consensus.cc:504] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:48:23.021970 17333 leader_election.cc:288] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949), 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841)
W20260926 14:48:23.022814 16929 leader_election.cc:341] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841): Not found: Tablet not found: 3dcf57495fdf4ec589b888b7d49f53cd
W20260926 14:48:23.022943 16929 leader_election.cc:341] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949): Not found: Tablet not found: 3dcf57495fdf4ec589b888b7d49f53cd
I20260926 14:48:23.022984 16929 leader_election.cc:302] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ae3d5b99ad84ef3b7d53e1062932d6f; no voters: 0f772a2f91c34cd494c6dae3a9faf35f, 7361b1493e19456896c3d5648556d40f
I20260926 14:48:23.023147 17333 raft_consensus.cc:2726] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:48:23.529651 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:23.529183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:33393)
0926 14:48:23.529485 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:23.529552 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":89,"threads_started":1}
I20260926 14:48:23.676317 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:23.675903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:40007)
0926 14:48:23.676157 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:23.676216 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":85,"threads_started":1}
I20260926 14:48:23.868022 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:23.867601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:36089)
0926 14:48:23.867865 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:23.867924 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260926 14:48:24.530870 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:24.530407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:53487)
0926 14:48:24.530668 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:24.530736 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":80,"threads_started":1}
I20260926 14:48:24.677522 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:24.677071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:50515)
0926 14:48:24.677361 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:24.677421 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":69,"threads_started":1}
I20260926 14:48:24.869284 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:24.868787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:55679)
0926 14:48:24.869117 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:24.869183 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":89,"threads_started":1}
I20260926 14:48:25.532078 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:25.531569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:41823)
0926 14:48:25.531884 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:25.531959 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":79,"threads_started":1}
I20260926 14:48:25.678690 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:25.678293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:51149)
0926 14:48:25.678531 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:25.678591 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":74,"threads_started":1}
I20260926 14:48:25.682538 17342 raft_consensus.cc:482] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:48:25.682657 17342 raft_consensus.cc:504] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:48:25.683156 17342 leader_election.cc:288] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949), 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841)
W20260926 14:48:25.683840 16929 leader_election.cc:341] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949): Not found: Tablet not found: 3dcf57495fdf4ec589b888b7d49f53cd
W20260926 14:48:25.683995 16929 leader_election.cc:341] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841): Not found: Tablet not found: 3dcf57495fdf4ec589b888b7d49f53cd
I20260926 14:48:25.684047 16929 leader_election.cc:302] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ae3d5b99ad84ef3b7d53e1062932d6f; no voters: 0f772a2f91c34cd494c6dae3a9faf35f, 7361b1493e19456896c3d5648556d40f
I20260926 14:48:25.684222 17342 raft_consensus.cc:2726] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:48:25.870522 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:25.870042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:47381)
0926 14:48:25.870350 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:25.870410 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":110,"threads_started":1}
I20260926 14:48:26.533283 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:26.532850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:46187)
0926 14:48:26.533123 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:26.533186 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":86,"threads_started":1}
I20260926 14:48:26.679889 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:26.679461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:47657)
0926 14:48:26.679729 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:26.679787 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
I20260926 14:48:26.871737 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:26.871276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:36555)
0926 14:48:26.871546 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:26.871609 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":79,"threads_started":1}
I20260926 14:48:27.534557 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:27.534092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:42249)
0926 14:48:27.534388 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:27.534455 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":77,"threads_started":1}
I20260926 14:48:27.681147 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:27.680709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:52579)
0926 14:48:27.680984 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:27.681046 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":98,"threads_started":1}
I20260926 14:48:27.872996 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:27.872564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:50273)
0926 14:48:27.872823 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:27.872894 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260926 14:48:28.535969 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:28.535429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:44487)
0926 14:48:28.535790 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:28.535861 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":75,"threads_started":1}
I20260926 14:48:28.682433 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:28.681944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:48987)
0926 14:48:28.682236 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:28.682317 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":77,"threads_started":1}
I20260926 14:48:28.874370 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:28.873871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:53975)
0926 14:48:28.874192 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:28.874259 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":94,"threads_started":1}
I20260926 14:48:29.537348 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:29.536819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:50707)
0926 14:48:29.537117 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:29.537186 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":80,"threads_started":1}
I20260926 14:48:29.683704 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:29.683244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:36121)
0926 14:48:29.683532 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:29.683599 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":78,"threads_started":1}
I20260926 14:48:29.875694 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:29.875233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:36911)
0926 14:48:29.875524 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:29.875586 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":95,"threads_started":1}
I20260926 14:48:30.538584 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:30.538152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:45173)
0926 14:48:30.538423 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:30.538484 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":77,"threads_started":1}
I20260926 14:48:30.684966 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:30.684537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:33281)
0926 14:48:30.684807 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:30.684866 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
I20260926 14:48:30.876941 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:30.876462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:47115)
0926 14:48:30.876777 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:30.876837 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":132,"threads_started":1}
I20260926 14:48:31.539929 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:31.539456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:35799)
0926 14:48:31.539762 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:31.539828 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":83,"threads_started":1}
I20260926 14:48:31.654820 17360 raft_consensus.cc:482] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:48:31.654978 17360 raft_consensus.cc:504] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f772a2f91c34cd494c6dae3a9faf35f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41949 } } peers { permanent_uuid: "7361b1493e19456896c3d5648556d40f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40841 } } peers { permanent_uuid: "9ae3d5b99ad84ef3b7d53e1062932d6f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43843 } }
I20260926 14:48:31.655468 17360 leader_election.cc:288] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949), 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841)
W20260926 14:48:31.656301 16929 leader_election.cc:341] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7361b1493e19456896c3d5648556d40f (127.0.98.2:40841): Not found: Tablet not found: 3dcf57495fdf4ec589b888b7d49f53cd
W20260926 14:48:31.656425 16929 leader_election.cc:341] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0f772a2f91c34cd494c6dae3a9faf35f (127.0.98.3:41949): Not found: Tablet not found: 3dcf57495fdf4ec589b888b7d49f53cd
I20260926 14:48:31.656472 16929 leader_election.cc:302] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ae3d5b99ad84ef3b7d53e1062932d6f; no voters: 0f772a2f91c34cd494c6dae3a9faf35f, 7361b1493e19456896c3d5648556d40f
I20260926 14:48:31.656657 17360 raft_consensus.cc:2726] T 3dcf57495fdf4ec589b888b7d49f53cd P 9ae3d5b99ad84ef3b7d53e1062932d6f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 14:48:31.686200 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:31.685738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:59929)
0926 14:48:31.686027 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:31.686088 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":96,"threads_started":1}
I20260926 14:48:31.878177 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:31.877708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.3:43469)
0926 14:48:31.878008 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:31.878072 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":93,"threads_started":1}
I20260926 14:48:32.541345 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:32.540908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.1:58317)
0926 14:48:32.541180 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:32.541243 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":88,"threads_started":1}
I20260926 14:48:32.687530 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:32.687103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35535 (local address 127.0.98.2:47801)
0926 14:48:32.687367 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:32.687426 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35535: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":92,"threads_started":1}
I20260926 14:48:32.860651 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 16873
I20260926 14:48:32.865967 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 17045
I20260926 14:48:32.870829 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 17180
2026-09-26T14:48:32Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59176 ms)
[----------] 30 tests from SecurityITest (240434 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:32 dist-test-slave-s7k6 krb5kdc[17367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:32 dist-test-slave-s7k6 krb5kdc[17367](info): set up 2 sockets
Sep 26 14:48:32 dist-test-slave-s7k6 krb5kdc[17367](info): commencing operation
krb5kdc: starting...
W20260926 14:48:33.826643 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:33 dist-test-slave-s7k6 krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:48:33.946405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38417
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:34.063587 17380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:34.063928 17380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:34.063987 17380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:34.068351 17380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:34.068439 17380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:34.068466 17380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:34.068488 17380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:34.068522 17380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:34.074384 17380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38417
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:34.075608 17380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:34.076697 17380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:34.085834 17388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:34.085834 17385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:34.085863 17386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:34.086038 17380 server_base.cc:1034] running on GCE node
I20260926 14:48:34.086869 17380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:34.088196 17380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:34.089385 17380 hybrid_clock.cc:648] HybridClock initialized: now 1790434114089356 us; error 43 us; skew 500 ppm
Sep 26 14:48:34 dist-test-slave-s7k6 krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:34.092501 17380 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:48:34.093537 17380 webserver.cc:460] Webserver started at http://127.0.98.62:39583/ using document root <none> and password file <none>
I20260926 14:48:34.094659 17380 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:34.094728 17380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:34.094978 17380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:34.097208 17380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "796a819ac8e346198c6f99a202ff4e4c"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "8a99c114bbb03da6e8571c6f7fd4e87b"
server_key_iv: "5797f4bcd1134a0c7bf73259ce5984ad"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.097810 17380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "796a819ac8e346198c6f99a202ff4e4c"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "8a99c114bbb03da6e8571c6f7fd4e87b"
server_key_iv: "5797f4bcd1134a0c7bf73259ce5984ad"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.102159 17380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260926 14:48:34.105103 17395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:34.106560 17380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:48:34.106693 17380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "796a819ac8e346198c6f99a202ff4e4c"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "8a99c114bbb03da6e8571c6f7fd4e87b"
server_key_iv: "5797f4bcd1134a0c7bf73259ce5984ad"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.106909 17380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:34.125833 17380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:34.129426 17380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:34.129626 17380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:34.136933 17380 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38417
I20260926 14:48:34.136955 17447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38417 every 8 connection(s)
I20260926 14:48:34.138011 17380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:34.141296 17448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:34.143111 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 17380
I20260926 14:48:34.143229 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:34.143528 392 external_mini_cluster.cc:1428] Setting key a0b3eb3e919a178cc27d364555fec251
I20260926 14:48:34.148460 17448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c: Bootstrap starting.
I20260926 14:48:34.151005 17448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:34.151912 17448 log.cc:824] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:48:34 dist-test-slave-s7k6 krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434113, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:48:34.154100 17448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c: No bootstrap required, opened a new log
I20260926 14:48:34.158413 17448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a819ac8e346198c6f99a202ff4e4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260926 14:48:34.158674 17448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:34.158720 17448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796a819ac8e346198c6f99a202ff4e4c, State: Initialized, Role: FOLLOWER
I20260926 14:48:34.159168 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.145139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39438 (local address 127.0.98.62:38417)
0926 14:48:34.145643 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:34.145659 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:34.145700 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:48:34.146444 (+ 744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:34.146467 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:34.146970 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:34.147338 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:34.148242 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.149324 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.150260 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.150816 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.150996 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.153678 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:34.153684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:34.153699 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:34.153728 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:34.155530 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:34.156158 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:34.156164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:34.156170 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:34.156302 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:34.156689 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:34.156692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:34.156693 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:34.157211 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:34.157383 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:34.157776 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 14:48:34.158159 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":156,"threads_started":1}
I20260926 14:48:34.159258 17448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a819ac8e346198c6f99a202ff4e4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260926 14:48:34.159441 17448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:34.159499 17448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:34.159579 17448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:34.160594 17448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a819ac8e346198c6f99a202ff4e4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260926 14:48:34.160977 17448 leader_election.cc:302] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 796a819ac8e346198c6f99a202ff4e4c; no voters:
I20260926 14:48:34.161291 17448 leader_election.cc:288] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:34.161382 17453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:34.161550 17453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [term 1 LEADER]: Becoming Leader. State: Replica: 796a819ac8e346198c6f99a202ff4e4c, State: Running, Role: LEADER
I20260926 14:48:34.161948 17453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a819ac8e346198c6f99a202ff4e4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } }
I20260926 14:48:34.162717 17448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:34.163942 17455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "796a819ac8e346198c6f99a202ff4e4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a819ac8e346198c6f99a202ff4e4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260926 14:48:34.163968 17454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 796a819ac8e346198c6f99a202ff4e4c. Latest consensus state: current_term: 1 leader_uuid: "796a819ac8e346198c6f99a202ff4e4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "796a819ac8e346198c6f99a202ff4e4c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38417 } } }
I20260926 14:48:34.164146 17454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:34.164485 17455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:34.164610 17462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:34.168105 17462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:34.173952 17462 catalog_manager.cc:1347] Generated new cluster ID: b7aeda7375484ee495f7ea2bfe8830ef
I20260926 14:48:34.174026 17462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:34.208784 17462 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:34.209791 17462 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:34.216686 17462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c: Generated new TSK 0
I20260926 14:48:34.217638 17462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:48:34.252261 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:34.388470 17474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:34.388754 17474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:34.388837 17474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:34.393102 17474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:34.393172 17474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:34.393255 17474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:48:34.398932 17474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:34.400072 17474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:34.401151 17474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:34.411147 17480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:34.411206 17482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:34.411223 17479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:34.411818 17474 server_base.cc:1034] running on GCE node
I20260926 14:48:34.412273 17474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:34.412910 17474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:34.414076 17474 hybrid_clock.cc:648] HybridClock initialized: now 1790434114414042 us; error 47 us; skew 500 ppm
Sep 26 14:48:34 dist-test-slave-s7k6 krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:34.417104 17474 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:48:34.418104 17474 webserver.cc:460] Webserver started at http://127.0.98.1:43027/ using document root <none> and password file <none>
I20260926 14:48:34.418711 17474 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:34.418816 17474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:34.419027 17474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:34.421324 17474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "7cc873c29bef4b2a9ff528c98605944c"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "7dc8e1b67674369fc704eb20088b43ac"
server_key_iv: "b34480d83b63d419cf321e433b16d687"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.421974 17474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7cc873c29bef4b2a9ff528c98605944c"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "7dc8e1b67674369fc704eb20088b43ac"
server_key_iv: "b34480d83b63d419cf321e433b16d687"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.426260 17474 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260926 14:48:34.429188 17489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:34.430935 17474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:34.431071 17474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "7cc873c29bef4b2a9ff528c98605944c"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "7dc8e1b67674369fc704eb20088b43ac"
server_key_iv: "b34480d83b63d419cf321e433b16d687"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.431212 17474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:34.448889 17474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:34.452332 17474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:34.452524 17474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:34.453151 17474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:34.454071 17474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:34.454121 17474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:34.454182 17474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:34.454216 17474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:34.463586 17474 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37993
I20260926 14:48:34.463620 17602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37993 every 8 connection(s)
I20260926 14:48:34.464617 17474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:48:34.469643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 17474
I20260926 14:48:34.469772 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:48:34.470041 392 external_mini_cluster.cc:1428] Setting key 57e2cb9c5c5e1cb5ed2ec10a22a16986
Sep 26 14:48:34 dist-test-slave-s7k6 krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 14:48:34.480399 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.466572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60515 (local address 127.0.98.62:38417)
0926 14:48:34.466707 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:34.466712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:34.467760 (+ 1048us) server_negotiation.cc:407] Connection header received
0926 14:48:34.469128 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:34.469132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:34.469234 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:34.469309 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:34.470942 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.471539 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.473272 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.473518 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.473609 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.476662 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:34.476674 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:34.476676 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:34.476705 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:34.478114 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:34.478687 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:34.478690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:34.478692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:34.478807 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:34.479161 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:34.479165 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:34.479167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:34.479363 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:34.479457 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:34.480107 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 14:48:34.480261 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:48:34.481180 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.466981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.1:60515)
0926 14:48:34.467536 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:34.467585 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:34.468888 (+ 1303us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:34.469552 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:34.469560 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:34.469983 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:34.470633 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.470653 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.471682 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.471686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.473095 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.473108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.473703 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.473708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.473933 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.474631 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:48:34.474652 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:34.476452 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:34.478254 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:34.478261 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:34.478281 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:34.478571 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:34.478900 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:34.478903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:34.478905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:34.479061 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:34.479484 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:34.479492 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:34.479811 (+ 319us) client_negotiation.cc:769] Sending connection context
0926 14:48:34.479994 (+ 183us) client_negotiation.cc:240] Negotiation successful
0926 14:48:34.480314 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 14:48:34.482539 17603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260926 14:48:34.482842 17603 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:34.483420 17603 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260926 14:48:34.485078 17412 ts_manager.cc:194] Registered new tserver with Master: 7cc873c29bef4b2a9ff528c98605944c (127.0.98.1:37993)
I20260926 14:48:34.486275 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:48:34.498911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:34.615823 17608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:34.616127 17608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:34.616173 17608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:34.620402 17608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:34.620489 17608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:34.620554 17608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:48:34.627453 17608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:34.628733 17608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:34.629875 17608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:34.640609 17614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:34.640834 17616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:34.640856 17608 server_base.cc:1034] running on GCE node
W20260926 14:48:34.640972 17613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:34.641490 17608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:34.642241 17608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:34.643426 17608 hybrid_clock.cc:648] HybridClock initialized: now 1790434114643389 us; error 55 us; skew 500 ppm
Sep 26 14:48:34 dist-test-slave-s7k6 krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:34.646378 17608 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:48:34.647442 17608 webserver.cc:460] Webserver started at http://127.0.98.2:41949/ using document root <none> and password file <none>
I20260926 14:48:34.648069 17608 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:34.648144 17608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:34.648383 17608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:34.650542 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "e05587877eda400b986410645896aaaa"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "a269f78d67d5d66d5514b0feb2438565"
server_key_iv: "d2930c0ee05a00697d39a1ed850ad24b"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.651149 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e05587877eda400b986410645896aaaa"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "a269f78d67d5d66d5514b0feb2438565"
server_key_iv: "d2930c0ee05a00697d39a1ed850ad24b"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.655149 17608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:48:34.657781 17623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:34.659262 17608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:34.659373 17608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "e05587877eda400b986410645896aaaa"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "a269f78d67d5d66d5514b0feb2438565"
server_key_iv: "d2930c0ee05a00697d39a1ed850ad24b"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.659482 17608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:34.674459 17608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:34.677918 17608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:34.678093 17608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:34.678656 17608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:34.679549 17608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:34.679600 17608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:34.679641 17608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:34.679654 17608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:34.688179 17608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36381
I20260926 14:48:34.688206 17736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36381 every 8 connection(s)
I20260926 14:48:34.689180 17608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:48:34.695756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 17608
I20260926 14:48:34.695866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:48:34.696132 392 external_mini_cluster.cc:1428] Setting key 8843dda74dfffc477f3e9ad49869af4f
Sep 26 14:48:34 dist-test-slave-s7k6 krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:48:34.704280 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.691243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44027 (local address 127.0.98.62:38417)
0926 14:48:34.691392 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:34.691396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:34.692303 (+ 907us) server_negotiation.cc:407] Connection header received
0926 14:48:34.693721 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:34.693725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:34.693783 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:34.693859 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:34.695297 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.695853 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.697299 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.697588 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.697755 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.700190 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:34.700194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:34.700196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:34.700227 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:34.701974 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:34.702581 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:34.702584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:34.702585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:34.702676 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:34.703054 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:34.703057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:34.703059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:34.703263 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:34.703357 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:34.703967 (+ 610us) server_negotiation.cc:299] Negotiation successful
0926 14:48:34.704100 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 14:48:34.705196 17739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.691657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.2:44027)
0926 14:48:34.692127 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:34.692187 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:34.693474 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:34.694069 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:34.694077 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:34.694482 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:34.695132 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.695150 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.695985 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.695988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.697157 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.697168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.697754 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.697757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.697911 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.698496 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:48:34.698512 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:34.700001 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:34.702128 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:34.702134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:34.702149 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:34.702450 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:34.702797 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:34.702800 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:34.702802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:34.702960 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:34.703368 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:34.703376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:34.703709 (+ 333us) client_negotiation.cc:769] Sending connection context
0926 14:48:34.703888 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 14:48:34.704264 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":112,"threads_started":1}
I20260926 14:48:34.706470 17737 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260926 14:48:34.706717 17737 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:34.707274 17737 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:48:34.708381 17412 ts_manager.cc:194] Registered new tserver with Master: e05587877eda400b986410645896aaaa (127.0.98.2:36381)
I20260926 14:48:34.709005 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44027
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:48:34.724229 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:34.841636 17742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:34.841925 17742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:34.841970 17742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:34.846148 17742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:34.846222 17742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:34.846285 17742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:48:34.852082 17742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38417
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:34.853206 17742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:34.854339 17742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:34.864262 17750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:34.864264 17747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:34.864262 17748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:34.864681 17742 server_base.cc:1034] running on GCE node
I20260926 14:48:34.865211 17742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:34.865960 17742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:34.867208 17742 hybrid_clock.cc:648] HybridClock initialized: now 1790434114867170 us; error 60 us; skew 500 ppm
Sep 26 14:48:34 dist-test-slave-s7k6 krb5kdc[17367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:34.870589 17742 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:48:34.871668 17742 webserver.cc:460] Webserver started at http://127.0.98.3:44525/ using document root <none> and password file <none>
I20260926 14:48:34.872319 17742 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:34.872390 17742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:34.872618 17742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:34.875001 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "47cccc88ed094fd99041baf36a5cafb5"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "efd2b35647c3468cfd657c810b104074"
server_key_iv: "dc97f889402d33a989cfe9d9e2dd312a"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.875608 17742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47cccc88ed094fd99041baf36a5cafb5"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "efd2b35647c3468cfd657c810b104074"
server_key_iv: "dc97f889402d33a989cfe9d9e2dd312a"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.879768 17742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:48:34.882596 17757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:34.884006 17742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:48:34.884130 17742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "47cccc88ed094fd99041baf36a5cafb5"
format_stamp: "Formatted at 2026-09-26 14:48:34 on dist-test-slave-s7k6"
server_key: "efd2b35647c3468cfd657c810b104074"
server_key_iv: "dc97f889402d33a989cfe9d9e2dd312a"
server_key_version: "encryptionkey@0"
I20260926 14:48:34.884254 17742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:34.910241 17742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:34.913766 17742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:34.913949 17742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:34.914584 17742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:34.915541 17742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:34.915598 17742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:34.915645 17742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:34.915674 17742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:34.924520 17742 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38653
I20260926 14:48:34.924633 17870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38653 every 8 connection(s)
I20260926 14:48:34.925648 17742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:48:34.931187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 17742
I20260926 14:48:34.931330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:48:34.931648 392 external_mini_cluster.cc:1428] Setting key c5f8997c6de96ca6d74f56ab213a6a5e
Sep 26 14:48:34 dist-test-slave-s7k6 krb5kdc[17367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:48:34.940286 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.927675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48079 (local address 127.0.98.62:38417)
0926 14:48:34.927831 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:34.927836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:34.928767 (+ 931us) server_negotiation.cc:407] Connection header received
0926 14:48:34.930068 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:34.930071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:34.930151 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:34.930237 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:34.931959 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.932687 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.934205 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.934426 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.934507 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.937119 (+ 2612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:34.937123 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:34.937125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:34.937153 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:34.938355 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:34.938854 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:34.938856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:34.938857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:34.938931 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:34.939219 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:34.939221 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:34.939222 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:34.939408 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:34.939479 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:34.940016 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 14:48:34.940154 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 14:48:34.941165 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.928085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38417 (local address 127.0.98.3:48079)
0926 14:48:34.928562 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:34.928658 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:34.929851 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:34.930540 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:34.930549 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:34.931062 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:34.931794 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.931815 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.932828 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.932832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.934097 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.934109 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.934551 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.934553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.934746 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.935400 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:48:34.935420 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:34.936947 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:34.938468 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:34.938474 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:34.938493 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:34.938776 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:34.939000 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:34.939002 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:34.939003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:34.939144 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:34.939491 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:34.939496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:34.939794 (+ 298us) client_negotiation.cc:769] Sending connection context
0926 14:48:34.939951 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 14:48:34.940241 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
I20260926 14:48:34.942435 17871 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38417
I20260926 14:48:34.942710 17871 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:34.943313 17871 heartbeater.cc:511] Master 127.0.98.62:38417 requested a full tablet report, sending...
I20260926 14:48:34.944474 17412 ts_manager.cc:194] Registered new tserver with Master: 47cccc88ed094fd99041baf36a5cafb5 (127.0.98.3:38653)
I20260926 14:48:34.945145 17412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48079
I20260926 14:48:34.945888 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:48:34.955268 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.948170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.0.98.62:38417)
0926 14:48:34.948314 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:34.948318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:34.948384 (+ 66us) server_negotiation.cc:407] Connection header received
0926 14:48:34.948493 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:34.948495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:34.948533 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:34.948598 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:34.949281 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.949796 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.950677 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.950961 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.951078 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.952437 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:34.952441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:34.952443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:34.952468 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:34.953627 (+ 1159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:34.954033 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:34.954035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:34.954037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:34.954127 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:34.954387 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:34.954389 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:34.954390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:34.954596 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:34.954692 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:34.954997 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 14:48:34.955110 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 14:48:34.958352 17411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56892:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:48:34.961520 17411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:48:34.976275 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.970741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.0.98.1:37993)
0926 14:48:34.971206 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:34.971224 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:34.971249 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:48:34.971331 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:34.971336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:34.971612 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:34.971830 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:34.972343 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.973243 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.974926 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.975542 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.975649 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.975828 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:34.976009 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 14:48:34.976116 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":83,"threads_started":1}
I20260926 14:48:34.976292 17882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.970549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37993 (local address 127.0.0.1:36108)
0926 14:48:34.970927 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:34.970982 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:34.971116 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:34.972023 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:34.972030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:34.972051 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:34.972201 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.972207 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.973371 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.973374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.974811 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.974821 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.975669 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.975673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.975804 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.975844 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 14:48:34.975961 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 14:48:34.976099 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":65,"threads_started":1}
I20260926 14:48:34.977211 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.971324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38653 (local address 127.0.0.1:58864)
0926 14:48:34.971769 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:34.971785 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:34.971880 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:34.973033 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:34.973036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:34.973056 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:34.973160 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.973166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.974453 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.974456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.975616 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.975627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.976797 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.976801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.976911 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.976936 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:48:34.976996 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 14:48:34.977072 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":22,"thread_start_us":96,"threads_started":1}
I20260926 14:48:34.977671 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.971769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36381 (local address 127.0.0.1:51112)
0926 14:48:34.972214 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:34.972224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:34.972304 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:34.973226 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:34.973229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:34.973248 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:34.973351 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.973358 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.974848 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.974851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.975985 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:34.975995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.977276 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.977281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:34.977374 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.977396 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:48:34.977463 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 14:48:34.977540 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":1,"thread_start_us":96,"threads_started":1}
I20260926 14:48:34.978137 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.971483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.0.98.3:38653)
0926 14:48:34.972550 (+ 1067us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:34.972561 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:34.972586 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:48:34.972675 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:34.972680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:34.972881 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:34.973044 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:34.973258 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.974315 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.975759 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.976631 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.976798 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.977816 (+ 1018us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:34.977876 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 14:48:34.977984 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":895,"thread_start_us":194,"threads_started":1}
I20260926 14:48:34.978180 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:34.972227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.0.98.2:36381)
0926 14:48:34.972636 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:34.972646 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:34.972664 (+ 18us) server_negotiation.cc:407] Connection header received
0926 14:48:34.972725 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:34.972729 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:34.972880 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:34.973044 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:34.973500 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.974659 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.976133 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:34.976967 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:34.977069 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:34.977219 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:34.977950 (+ 731us) server_negotiation.cc:299] Negotiation successful
0926 14:48:34.978046 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":110,"threads_started":1}
I20260926 14:48:34.980176 17537 tablet_service.cc:1467] Processing CreateTablet for tablet e27e15d9d1594d01bd35c028477f66a8 (DEFAULT_TABLE table=test-table [id=875fd3c77e29478786a4b6c361cb7f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:34.981747 17537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27e15d9d1594d01bd35c028477f66a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:34.981328 17805 tablet_service.cc:1467] Processing CreateTablet for tablet e27e15d9d1594d01bd35c028477f66a8 (DEFAULT_TABLE table=test-table [id=875fd3c77e29478786a4b6c361cb7f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:34.982522 17805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27e15d9d1594d01bd35c028477f66a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:34.983353 17671 tablet_service.cc:1467] Processing CreateTablet for tablet e27e15d9d1594d01bd35c028477f66a8 (DEFAULT_TABLE table=test-table [id=875fd3c77e29478786a4b6c361cb7f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:34.984288 17671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e27e15d9d1594d01bd35c028477f66a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:34.989830 17888 tablet_bootstrap.cc:492] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c: Bootstrap starting.
I20260926 14:48:34.990688 17890 tablet_bootstrap.cc:492] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa: Bootstrap starting.
I20260926 14:48:34.991448 17889 tablet_bootstrap.cc:492] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5: Bootstrap starting.
I20260926 14:48:34.992643 17888 tablet_bootstrap.cc:654] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:34.992879 17890 tablet_bootstrap.cc:654] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:34.993574 17889 tablet_bootstrap.cc:654] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:34.993582 17888 log.cc:824] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:34.994448 17890 log.cc:824] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:34.994715 17889 log.cc:824] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:34.995949 17888 tablet_bootstrap.cc:492] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c: No bootstrap required, opened a new log
I20260926 14:48:34.996178 17888 ts_tablet_manager.cc:1397] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260926 14:48:34.996498 17889 tablet_bootstrap.cc:492] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5: No bootstrap required, opened a new log
I20260926 14:48:34.996572 17890 tablet_bootstrap.cc:492] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa: No bootstrap required, opened a new log
I20260926 14:48:34.996675 17889 ts_tablet_manager.cc:1397] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:48:34.996763 17890 ts_tablet_manager.cc:1397] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260926 14:48:34.999828 17889 raft_consensus.cc:348] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc873c29bef4b2a9ff528c98605944c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37993 } } peers { permanent_uuid: "47cccc88ed094fd99041baf36a5cafb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } } peers { permanent_uuid: "e05587877eda400b986410645896aaaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36381 } }
I20260926 14:48:35.000115 17889 raft_consensus.cc:374] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:35.000171 17889 raft_consensus.cc:729] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cccc88ed094fd99041baf36a5cafb5, State: Initialized, Role: FOLLOWER
I20260926 14:48:35.000629 17889 consensus_queue.cc:260] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc873c29bef4b2a9ff528c98605944c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37993 } } peers { permanent_uuid: "47cccc88ed094fd99041baf36a5cafb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } } peers { permanent_uuid: "e05587877eda400b986410645896aaaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36381 } }
I20260926 14:48:35.000646 17888 raft_consensus.cc:348] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc873c29bef4b2a9ff528c98605944c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37993 } } peers { permanent_uuid: "47cccc88ed094fd99041baf36a5cafb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } } peers { permanent_uuid: "e05587877eda400b986410645896aaaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36381 } }
I20260926 14:48:35.000646 17890 raft_consensus.cc:348] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc873c29bef4b2a9ff528c98605944c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37993 } } peers { permanent_uuid: "47cccc88ed094fd99041baf36a5cafb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } } peers { permanent_uuid: "e05587877eda400b986410645896aaaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36381 } }
I20260926 14:48:35.000916 17888 raft_consensus.cc:374] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:35.000916 17890 raft_consensus.cc:374] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:35.000982 17890 raft_consensus.cc:729] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e05587877eda400b986410645896aaaa, State: Initialized, Role: FOLLOWER
I20260926 14:48:35.000982 17888 raft_consensus.cc:729] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cc873c29bef4b2a9ff528c98605944c, State: Initialized, Role: FOLLOWER
I20260926 14:48:35.001438 17888 consensus_queue.cc:260] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc873c29bef4b2a9ff528c98605944c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37993 } } peers { permanent_uuid: "47cccc88ed094fd99041baf36a5cafb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } } peers { permanent_uuid: "e05587877eda400b986410645896aaaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36381 } }
I20260926 14:48:35.001439 17890 consensus_queue.cc:260] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc873c29bef4b2a9ff528c98605944c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37993 } } peers { permanent_uuid: "47cccc88ed094fd99041baf36a5cafb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } } peers { permanent_uuid: "e05587877eda400b986410645896aaaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36381 } }
I20260926 14:48:35.001626 17889 ts_tablet_manager.cc:1428] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:48:35.001989 17871 heartbeater.cc:503] Master 127.0.98.62:38417 was elected leader, sending a full tablet report...
I20260926 14:48:35.002399 17890 ts_tablet_manager.cc:1428] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260926 14:48:35.002404 17888 ts_tablet_manager.cc:1428] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 14:48:35.002786 17737 heartbeater.cc:503] Master 127.0.98.62:38417 was elected leader, sending a full tablet report...
I20260926 14:48:35.003120 17603 heartbeater.cc:503] Master 127.0.98.62:38417 was elected leader, sending a full tablet report...
I20260926 14:48:35.038918 17896 raft_consensus.cc:482] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:48:35.039131 17896 raft_consensus.cc:504] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc873c29bef4b2a9ff528c98605944c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37993 } } peers { permanent_uuid: "47cccc88ed094fd99041baf36a5cafb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } } peers { permanent_uuid: "e05587877eda400b986410645896aaaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36381 } }
I20260926 14:48:35.040301 17896 leader_election.cc:288] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47cccc88ed094fd99041baf36a5cafb5 (127.0.98.3:38653), e05587877eda400b986410645896aaaa (127.0.98.2:36381)
I20260926 14:48:35.045841 17887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.041199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48095 (local address 127.0.98.2:36381)
0926 14:48:35.041323 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:35.041327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:35.041400 (+ 73us) server_negotiation.cc:407] Connection header received
0926 14:48:35.041507 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:35.041510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:35.041555 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:35.041627 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:35.041864 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.042659 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.044510 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.045190 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.045312 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.045388 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:35.045634 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 14:48:35.045711 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:48:35.045866 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.041021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36381 (local address 127.0.98.1:48095)
0926 14:48:35.041319 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:35.041330 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:35.041419 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:35.041644 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:35.041646 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:35.041659 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:35.041767 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:35.041771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.042779 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.042782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:35.044357 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:35.044370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.045343 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.045346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:35.045483 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.045512 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 14:48:35.045596 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:48:35.045679 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":127,"thread_start_us":90,"threads_started":1}
I20260926 14:48:35.045921 17885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.040655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60669 (local address 127.0.98.3:38653)
0926 14:48:35.040809 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:35.040814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:35.041074 (+ 260us) server_negotiation.cc:407] Connection header received
0926 14:48:35.041258 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:35.041262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:35.041318 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:35.041369 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:35.041792 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.042625 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.044508 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.045296 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.045421 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.045471 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:35.045738 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 14:48:35.045813 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:48:35.045876 17897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.040591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38653 (local address 127.0.98.1:60669)
0926 14:48:35.040940 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:35.040963 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:35.041115 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:35.041534 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:35.041537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:35.041555 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:35.041687 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:35.041695 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.042778 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.042782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:35.044357 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:35.044370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.045515 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.045519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:35.045626 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.045646 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 14:48:35.045709 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 14:48:35.045774 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":93,"thread_start_us":133,"threads_started":1}
I20260926 14:48:35.046677 17691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27e15d9d1594d01bd35c028477f66a8" candidate_uuid: "7cc873c29bef4b2a9ff528c98605944c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e05587877eda400b986410645896aaaa" is_pre_election: true
I20260926 14:48:35.047024 17825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27e15d9d1594d01bd35c028477f66a8" candidate_uuid: "7cc873c29bef4b2a9ff528c98605944c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cccc88ed094fd99041baf36a5cafb5" is_pre_election: true
I20260926 14:48:35.047102 17691 raft_consensus.cc:2454] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cc873c29bef4b2a9ff528c98605944c in term 0.
I20260926 14:48:35.047343 17825 raft_consensus.cc:2454] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cc873c29bef4b2a9ff528c98605944c in term 0.
I20260926 14:48:35.047693 17491 leader_election.cc:302] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cc873c29bef4b2a9ff528c98605944c, e05587877eda400b986410645896aaaa; no voters:
I20260926 14:48:35.047999 17896 raft_consensus.cc:2781] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:48:35.048089 17896 raft_consensus.cc:482] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:48:35.048122 17896 raft_consensus.cc:3037] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:35.049173 17896 raft_consensus.cc:504] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc873c29bef4b2a9ff528c98605944c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37993 } } peers { permanent_uuid: "47cccc88ed094fd99041baf36a5cafb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } } peers { permanent_uuid: "e05587877eda400b986410645896aaaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36381 } }
I20260926 14:48:35.049518 17896 leader_election.cc:288] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [CANDIDATE]: Term 1 election: Requested vote from peers 47cccc88ed094fd99041baf36a5cafb5 (127.0.98.3:38653), e05587877eda400b986410645896aaaa (127.0.98.2:36381)
I20260926 14:48:35.049852 17825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27e15d9d1594d01bd35c028477f66a8" candidate_uuid: "7cc873c29bef4b2a9ff528c98605944c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cccc88ed094fd99041baf36a5cafb5"
I20260926 14:48:35.049914 17691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27e15d9d1594d01bd35c028477f66a8" candidate_uuid: "7cc873c29bef4b2a9ff528c98605944c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e05587877eda400b986410645896aaaa"
I20260926 14:48:35.049988 17825 raft_consensus.cc:3037] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:35.050045 17691 raft_consensus.cc:3037] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:35.051121 17825 raft_consensus.cc:2454] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cc873c29bef4b2a9ff528c98605944c in term 1.
I20260926 14:48:35.051182 17691 raft_consensus.cc:2454] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7cc873c29bef4b2a9ff528c98605944c in term 1.
I20260926 14:48:35.051437 17491 leader_election.cc:302] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47cccc88ed094fd99041baf36a5cafb5, 7cc873c29bef4b2a9ff528c98605944c; no voters:
I20260926 14:48:35.051604 17896 raft_consensus.cc:2781] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:35.051829 17896 raft_consensus.cc:686] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [term 1 LEADER]: Becoming Leader. State: Replica: 7cc873c29bef4b2a9ff528c98605944c, State: Running, Role: LEADER
I20260926 14:48:35.052194 17896 consensus_queue.cc:237] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc873c29bef4b2a9ff528c98605944c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37993 } } peers { permanent_uuid: "47cccc88ed094fd99041baf36a5cafb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } } peers { permanent_uuid: "e05587877eda400b986410645896aaaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36381 } }
I20260926 14:48:35.055365 17412 catalog_manager.cc:5495] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c reported cstate change: term changed from 0 to 1, leader changed from <none> to 7cc873c29bef4b2a9ff528c98605944c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "7cc873c29bef4b2a9ff528c98605944c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cc873c29bef4b2a9ff528c98605944c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47cccc88ed094fd99041baf36a5cafb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e05587877eda400b986410645896aaaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36381 } health_report { overall_health: UNKNOWN } } }
I20260926 14:48:35.075541 17883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.072189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.0.98.1:37993)
0926 14:48:35.072318 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:35.072322 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:35.072337 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:48:35.072430 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:35.072432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:35.072497 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:35.072590 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:48:35.072803 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.073302 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.074168 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.074383 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.074474 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.074711 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:48:35.075202 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:48:35.075293 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:35.075367 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 14:48:35.075422 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:48:35.082108 17825 raft_consensus.cc:1261] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5 [term 1 FOLLOWER]: Refusing update from remote peer 7cc873c29bef4b2a9ff528c98605944c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:35.082108 17691 raft_consensus.cc:1261] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa [term 1 FOLLOWER]: Refusing update from remote peer 7cc873c29bef4b2a9ff528c98605944c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:35.082967 17896 consensus_queue.cc:1035] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e05587877eda400b986410645896aaaa" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:48:35.083204 17899 consensus_queue.cc:1035] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c [LEADER]: Connected to new peer: Peer: permanent_uuid: "47cccc88ed094fd99041baf36a5cafb5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:48:35.101020 17410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:35.101197 17410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56892:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:35.104030 17410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 796a819ac8e346198c6f99a202ff4e4c: Sending DeleteTablet for 3 replicas of tablet e27e15d9d1594d01bd35c028477f66a8
I20260926 14:48:35.105110 17537 tablet_service.cc:1514] Processing DeleteTablet for tablet e27e15d9d1594d01bd35c028477f66a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36108
I20260926 14:48:35.105149 17671 tablet_service.cc:1514] Processing DeleteTablet for tablet e27e15d9d1594d01bd35c028477f66a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51112
I20260926 14:48:35.105456 17805 tablet_service.cc:1514] Processing DeleteTablet for tablet e27e15d9d1594d01bd35c028477f66a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58864
I20260926 14:48:35.105880 17912 tablet_replica.cc:330] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa: stopping tablet replica
I20260926 14:48:35.106036 17913 tablet_replica.cc:330] T e27e15d9d1594d01bd35c028477f66a8 P 7cc873c29bef4b2a9ff528c98605944c: stopping tablet replica
I20260926 14:48:35.106166 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 17474
I20260926 14:48:35.107103 17914 tablet_replica.cc:330] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5: stopping tablet replica
I20260926 14:48:35.107467 17914 raft_consensus.cc:2229] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:48:35.107841 17914 raft_consensus.cc:2258] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:35.108004 17912 raft_consensus.cc:2229] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:48:35.108369 17912 raft_consensus.cc:2258] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:35.109514 17914 ts_tablet_manager.cc:1905] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:48:35.109625 17912 ts_tablet_manager.cc:1905] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:48:35.112139 17912 ts_tablet_manager.cc:1918] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:48:35.112237 17912 log.cc:1196] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/e27e15d9d1594d01bd35c028477f66a8
I20260926 14:48:35.112336 17914 ts_tablet_manager.cc:1918] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:48:35.112449 17914 log.cc:1196] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/e27e15d9d1594d01bd35c028477f66a8
I20260926 14:48:35.112591 17912 ts_tablet_manager.cc:1939] T e27e15d9d1594d01bd35c028477f66a8 P e05587877eda400b986410645896aaaa: Deleting consensus metadata
I20260926 14:48:35.112762 17914 ts_tablet_manager.cc:1939] T e27e15d9d1594d01bd35c028477f66a8 P 47cccc88ed094fd99041baf36a5cafb5: Deleting consensus metadata
I20260926 14:48:35.114123 17397 catalog_manager.cc:4841] TS 47cccc88ed094fd99041baf36a5cafb5 (127.0.98.3:38653): tablet e27e15d9d1594d01bd35c028477f66a8 (table test-table [id=875fd3c77e29478786a4b6c361cb7f52]) successfully deleted
W20260926 14:48:35.114418 17397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37993 (error 108)
I20260926 14:48:35.114599 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 17608
I20260926 14:48:35.116737 17397 catalog_manager.cc:4841] TS e05587877eda400b986410645896aaaa (127.0.98.2:36381): tablet e27e15d9d1594d01bd35c028477f66a8 (table test-table [id=875fd3c77e29478786a4b6c361cb7f52]) successfully deleted
I20260926 14:48:35.117041 17886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.116583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37993 (local address 127.0.0.1:36126)
0926 14:48:35.116774 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:35.116901 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 14:48:35.117210 17397 catalog_manager.cc:4584] TS 7cc873c29bef4b2a9ff528c98605944c (127.0.98.1:37993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e27e15d9d1594d01bd35c028477f66a8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37993: connect: Connection refused (error 111)
I20260926 14:48:35.122000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 17742
I20260926 14:48:35.127277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 17380
2026-09-26T14:48:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2256 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T14:48:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:35Z Disabled control of system clock
I20260926 14:48:35.156191 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33161
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:48:35.270007 17919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:35.270323 17919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:35.270367 17919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:35.274601 17919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:35.274679 17919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:35.274694 17919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:35.274709 17919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:35.274724 17919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:35.279875 17919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:35.281036 17919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:35.282541 17919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:35.291432 17927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:35.291436 17925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:35.291662 17919 server_base.cc:1034] running on GCE node
W20260926 14:48:35.291432 17924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:35.292595 17919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:35.293823 17919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:35.294986 17919 hybrid_clock.cc:648] HybridClock initialized: now 1790434115294967 us; error 33 us; skew 500 ppm
I20260926 14:48:35.297021 17919 webserver.cc:460] Webserver started at http://127.0.98.62:39705/ using document root <none> and password file <none>
I20260926 14:48:35.297627 17919 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:35.297699 17919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:35.297897 17919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:35.300122 17919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "702598b796b54bf3bc3620ce3eda2a6f"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "6970e08b33a8e2653a5b181ecece76ba"
server_key_iv: "030b67a6de3c521b2b8220e19fae5858"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.300705 17919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "702598b796b54bf3bc3620ce3eda2a6f"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "6970e08b33a8e2653a5b181ecece76ba"
server_key_iv: "030b67a6de3c521b2b8220e19fae5858"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.304973 17919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:48:35.307721 17933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:35.309141 17919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:35.309265 17919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "702598b796b54bf3bc3620ce3eda2a6f"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "6970e08b33a8e2653a5b181ecece76ba"
server_key_iv: "030b67a6de3c521b2b8220e19fae5858"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.309376 17919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:35.319378 17919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:35.320109 17919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:35.320268 17919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:35.327378 17919 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33161
I20260926 14:48:35.327437 17985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33161 every 8 connection(s)
I20260926 14:48:35.328572 17919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:35.332084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 17919
I20260926 14:48:35.332193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:35.332284 17986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:35.332494 392 external_mini_cluster.cc:1428] Setting key 435acaa11982c84f10713234e4e45c90
I20260926 14:48:35.339244 17986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f: Bootstrap starting.
I20260926 14:48:35.341818 17986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:35.342684 17986 log.cc:824] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:35.342998 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.334065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34498 (local address 127.0.98.62:33161)
0926 14:48:35.334864 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:35.334881 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:35.334922 (+ 41us) server_negotiation.cc:407] Connection header received
0926 14:48:35.335689 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:35.335709 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:35.336157 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:35.336560 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:35.337035 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.338035 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.339002 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.339614 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.339777 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.340827 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:35.340832 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:35.340847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:35.341366 (+ 519us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:35.341436 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:35.341460 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:35.341558 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:35.341720 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 14:48:35.342054 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":158,"threads_started":1}
I20260926 14:48:35.345077 17986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f: No bootstrap required, opened a new log
I20260926 14:48:35.348836 17986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702598b796b54bf3bc3620ce3eda2a6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } }
I20260926 14:48:35.349099 17986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:35.349134 17986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 702598b796b54bf3bc3620ce3eda2a6f, State: Initialized, Role: FOLLOWER
I20260926 14:48:35.349526 17986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702598b796b54bf3bc3620ce3eda2a6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } }
I20260926 14:48:35.349634 17986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:35.349680 17986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:35.349753 17986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:35.350679 17986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702598b796b54bf3bc3620ce3eda2a6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } }
I20260926 14:48:35.351040 17986 leader_election.cc:302] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 702598b796b54bf3bc3620ce3eda2a6f; no voters:
I20260926 14:48:35.351348 17986 leader_election.cc:288] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:35.351444 17991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:35.351626 17991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [term 1 LEADER]: Becoming Leader. State: Replica: 702598b796b54bf3bc3620ce3eda2a6f, State: Running, Role: LEADER
I20260926 14:48:35.351935 17991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702598b796b54bf3bc3620ce3eda2a6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } }
I20260926 14:48:35.352736 17986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:35.353708 17992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "702598b796b54bf3bc3620ce3eda2a6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702598b796b54bf3bc3620ce3eda2a6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } }
I20260926 14:48:35.353855 17992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:35.354214 17993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 702598b796b54bf3bc3620ce3eda2a6f. Latest consensus state: current_term: 1 leader_uuid: "702598b796b54bf3bc3620ce3eda2a6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "702598b796b54bf3bc3620ce3eda2a6f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33161 } } }
I20260926 14:48:35.354261 18000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:35.354332 17993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:35.357635 18000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:35.363803 18000 catalog_manager.cc:1347] Generated new cluster ID: e437ba43a28d4e749fefe21b2b2650fc
I20260926 14:48:35.363899 18000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:35.390081 18000 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:35.391517 18000 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:35.406842 18000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f: Generated new TSK 0
I20260926 14:48:35.407704 18000 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:35.418874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33161
--builtin_ntp_servers=127.0.98.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:48:35.540256 18010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:35.540555 18010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:35.540593 18010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:35.545014 18010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:35.545094 18010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:35.545154 18010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:48:35.550352 18010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:35.551621 18010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:35.553128 18010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:48:35.562999 18010 server_base.cc:1034] running on GCE node
W20260926 14:48:35.562960 18016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:35.563016 18015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:35.562960 18018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:35.563736 18010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:35.564406 18010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:35.565565 18010 hybrid_clock.cc:648] HybridClock initialized: now 1790434115565538 us; error 43 us; skew 500 ppm
I20260926 14:48:35.567608 18010 webserver.cc:460] Webserver started at http://127.0.98.1:34171/ using document root <none> and password file <none>
I20260926 14:48:35.568184 18010 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:35.568248 18010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:35.568460 18010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:35.570567 18010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "056463353d4b4388bcb5e9f1c86f4c13"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "b541f9c75b54e0c1d35098e5b64190bb"
server_key_iv: "cdb34cc0d745075e02a834fd47a81d03"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.571136 18010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "056463353d4b4388bcb5e9f1c86f4c13"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "b541f9c75b54e0c1d35098e5b64190bb"
server_key_iv: "cdb34cc0d745075e02a834fd47a81d03"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.575110 18010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:48:35.577656 18024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:35.579000 18010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:48:35.579114 18010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "056463353d4b4388bcb5e9f1c86f4c13"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "b541f9c75b54e0c1d35098e5b64190bb"
server_key_iv: "cdb34cc0d745075e02a834fd47a81d03"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.579234 18010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:35.593971 18010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:35.594717 18010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:35.594939 18010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:35.595558 18010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:35.596540 18010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:35.596596 18010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:35.596642 18010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:35.596671 18010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:35.605711 18010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34699
I20260926 14:48:35.605839 18137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34699 every 8 connection(s)
I20260926 14:48:35.606698 18010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:48:35.616412 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.608631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35229 (local address 127.0.98.62:33161)
0926 14:48:35.608807 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:35.608812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:35.609971 (+ 1159us) server_negotiation.cc:407] Connection header received
0926 14:48:35.611452 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:35.611455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:35.611506 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:35.611592 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:35.612549 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.613135 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.614413 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.614636 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.614723 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.615926 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:35.615929 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:35.615931 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:35.615943 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:35.615962 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:35.615970 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:35.616028 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:35.616217 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 14:48:35.616287 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260926 14:48:35.616717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 18010
I20260926 14:48:35.616845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:48:35.617159 392 external_mini_cluster.cc:1428] Setting key 9f6bd3ed717ecaebf97ab2cf9c6bba91
I20260926 14:48:35.617611 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.609109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33161 (local address 127.0.98.1:35229)
0926 14:48:35.609807 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:35.609852 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:35.611206 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:35.611794 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:35.611801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:35.611895 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:35.612427 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:35.612441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.613275 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.613278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:35.614291 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:35.614303 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.614848 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.614852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:35.615012 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.615658 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:48:35.615660 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:35.615706 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:48:35.615782 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:35.616073 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:35.616082 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:35.616087 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:48:35.616207 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 14:48:35.616481 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":111,"threads_started":1}
I20260926 14:48:35.619004 18138 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33161
I20260926 14:48:35.619261 18138 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:35.619467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33161
--builtin_ntp_servers=127.0.98.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:48:35.619882 18138 heartbeater.cc:511] Master 127.0.98.62:33161 requested a full tablet report, sending...
I20260926 14:48:35.621660 17950 ts_manager.cc:194] Registered new tserver with Master: 056463353d4b4388bcb5e9f1c86f4c13 (127.0.98.1:34699)
I20260926 14:48:35.623086 17950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35229
W20260926 14:48:35.734181 18141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:35.734491 18141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:35.734547 18141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:35.738710 18141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:35.738803 18141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:35.738880 18141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:48:35.743917 18141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:35.745136 18141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:35.746654 18141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:35.756671 18147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:35.756680 18149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:35.756676 18146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:35.757102 18141 server_base.cc:1034] running on GCE node
I20260926 14:48:35.757552 18141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:35.758219 18141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:35.759374 18141 hybrid_clock.cc:648] HybridClock initialized: now 1790434115759356 us; error 32 us; skew 500 ppm
I20260926 14:48:35.761260 18141 webserver.cc:460] Webserver started at http://127.0.98.2:43773/ using document root <none> and password file <none>
I20260926 14:48:35.761799 18141 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:35.761854 18141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:35.762033 18141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:35.764159 18141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "baf211539dc74bd8b6fd4da8c6e133ca"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "91f0703546becf25106f25f5ad26df07"
server_key_iv: "9b2371f90f940c5baacbb0f3eb4874f8"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.764688 18141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "baf211539dc74bd8b6fd4da8c6e133ca"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "91f0703546becf25106f25f5ad26df07"
server_key_iv: "9b2371f90f940c5baacbb0f3eb4874f8"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.768823 18141 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:48:35.771518 18155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:35.772866 18141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:35.772967 18141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "baf211539dc74bd8b6fd4da8c6e133ca"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "91f0703546becf25106f25f5ad26df07"
server_key_iv: "9b2371f90f940c5baacbb0f3eb4874f8"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.773085 18141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:35.785432 18141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:35.786109 18141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:35.786285 18141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:35.787042 18141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:35.788020 18141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:35.788065 18141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:35.788105 18141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:35.788118 18141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:35.796306 18141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43523
I20260926 14:48:35.796345 18268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43523 every 8 connection(s)
I20260926 14:48:35.797190 18141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:48:35.806176 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.798943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33211 (local address 127.0.98.62:33161)
0926 14:48:35.799096 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:35.799101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:35.800235 (+ 1134us) server_negotiation.cc:407] Connection header received
0926 14:48:35.801602 (+ 1367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:35.801605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:35.801658 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:35.801723 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:35.802652 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.803221 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.804400 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.804612 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.804694 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.805765 (+ 1071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:35.805768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:35.805770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:35.805782 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:35.805803 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:35.805811 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:35.805876 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:35.806002 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 14:48:35.806053 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:48:35.806464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 18141
I20260926 14:48:35.806615 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:48:35.806950 392 external_mini_cluster.cc:1428] Setting key bbda5a1f6c94e50f3a450fdf870cf52d
I20260926 14:48:35.807343 18271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.799388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33161 (local address 127.0.98.2:33211)
0926 14:48:35.800090 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:35.800133 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:35.801400 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:35.801872 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:35.801879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:35.801962 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:35.802498 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:35.802510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.803354 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.803357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:35.804298 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:35.804307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.804770 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.804774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:35.804917 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.805502 (+ 585us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:48:35.805504 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:35.805544 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:48:35.805609 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:35.805890 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:35.805897 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:35.805902 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 14:48:35.805996 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 14:48:35.806315 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260926 14:48:35.808323 18269 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33161
I20260926 14:48:35.808545 18269 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:35.808916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33161
--builtin_ntp_servers=127.0.98.20:46023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:48:35.809075 18269 heartbeater.cc:511] Master 127.0.98.62:33161 requested a full tablet report, sending...
I20260926 14:48:35.810050 17950 ts_manager.cc:194] Registered new tserver with Master: baf211539dc74bd8b6fd4da8c6e133ca (127.0.98.2:43523)
I20260926 14:48:35.810607 17950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:33211
W20260926 14:48:35.919378 18272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:35.919673 18272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:35.919711 18272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:35.923699 18272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:35.923759 18272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:35.923817 18272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:48:35.928660 18272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33161
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18272
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:35.929889 18272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:35.931319 18272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:35.940603 18280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:35.940603 18278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:35.940649 18277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:35.940831 18272 server_base.cc:1034] running on GCE node
I20260926 14:48:35.941447 18272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:35.942104 18272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:35.943267 18272 hybrid_clock.cc:648] HybridClock initialized: now 1790434115943239 us; error 46 us; skew 500 ppm
I20260926 14:48:35.945238 18272 webserver.cc:460] Webserver started at http://127.0.98.3:33083/ using document root <none> and password file <none>
I20260926 14:48:35.945811 18272 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:35.945871 18272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:35.946128 18272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:35.948258 18272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "d0056148643040789750a51e7df3bc95"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "e45d5bb6378a16ceef3bbe55c5898c51"
server_key_iv: "182f1689f0b04c325ed2904e8b9c8a32"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.948783 18272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d0056148643040789750a51e7df3bc95"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "e45d5bb6378a16ceef3bbe55c5898c51"
server_key_iv: "182f1689f0b04c325ed2904e8b9c8a32"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.952495 18272 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:48:35.955250 18286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:35.956535 18272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:35.956650 18272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "d0056148643040789750a51e7df3bc95"
format_stamp: "Formatted at 2026-09-26 14:48:35 on dist-test-slave-s7k6"
server_key: "e45d5bb6378a16ceef3bbe55c5898c51"
server_key_iv: "182f1689f0b04c325ed2904e8b9c8a32"
server_key_version: "encryptionkey@0"
I20260926 14:48:35.956745 18272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:35.972405 18272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:35.973109 18272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:35.973273 18272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:35.973860 18272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:35.974817 18272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:35.974875 18272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:35.974918 18272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:35.974937 18272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:35.983407 18272 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34689
I20260926 14:48:35.983434 18399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34689 every 8 connection(s)
I20260926 14:48:35.984400 18272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:48:35.985158 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 18272
I20260926 14:48:35.985258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:48:35.985539 392 external_mini_cluster.cc:1428] Setting key ce77719c1da03ce4c511947fefa3a67b
I20260926 14:48:35.993623 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.986375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33525 (local address 127.0.98.62:33161)
0926 14:48:35.986505 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:35.986509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:35.987841 (+ 1332us) server_negotiation.cc:407] Connection header received
0926 14:48:35.989232 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:35.989235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:35.989289 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:35.989357 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:35.990238 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.990790 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.991866 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.992097 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.992182 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.993179 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:35.993182 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:35.993185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:35.993198 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:35.993220 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:35.993227 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:35.993283 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:35.993413 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 14:48:35.993470 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 14:48:35.994438 18402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:35.987002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33161 (local address 127.0.98.3:33525)
0926 14:48:35.987719 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:35.987763 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:35.989036 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:35.989516 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:35.989523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:35.989615 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:35.990108 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:35.990120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.990889 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.990893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:35.991756 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:35.991765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:35.992228 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:35.992231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:35.992380 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:35.992922 (+ 542us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:48:35.992923 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:35.992961 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:48:35.993022 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:35.993292 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:35.993300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:35.993306 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:48:35.993408 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:48:35.993673 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":141,"threads_started":1}
I20260926 14:48:35.995445 18400 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33161
I20260926 14:48:35.995652 18400 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:35.996132 18400 heartbeater.cc:511] Master 127.0.98.62:33161 requested a full tablet report, sending...
I20260926 14:48:35.997126 17950 ts_manager.cc:194] Registered new tserver with Master: d0056148643040789750a51e7df3bc95 (127.0.98.3:34689)
I20260926 14:48:35.997658 17950 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33525
I20260926 14:48:35.999647 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:48:36.004765 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.001407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.0.98.62:33161)
0926 14:48:36.001529 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:36.001533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:36.001647 (+ 114us) server_negotiation.cc:407] Connection header received
0926 14:48:36.001751 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:36.001753 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:36.001788 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:36.001847 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:36.002081 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.002572 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.003379 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.003597 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.003669 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.004364 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:36.004367 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:36.004369 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:36.004379 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:36.004397 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:36.004404 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:36.004456 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:36.004590 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 14:48:36.004635 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:48:36.007462 17950 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:48:36.010406 17950 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:48:36.024663 18411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.018347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34699 (local address 127.0.0.1:35866)
0926 14:48:36.018827 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:36.018875 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:36.018996 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:36.020144 (+ 1148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:36.020146 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:36.020160 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:36.020241 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:36.020245 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.021519 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.021523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:36.022969 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:36.022980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.024130 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.024133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:36.024220 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.024254 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 14:48:36.024361 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 14:48:36.024495 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":104,"thread_start_us":83,"threads_started":1}
I20260926 14:48:36.024696 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.018660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.0.98.1:34699)
0926 14:48:36.019105 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:36.019116 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:36.019143 (+ 27us) server_negotiation.cc:407] Connection header received
0926 14:48:36.019638 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:36.019646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:36.019849 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:36.019983 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:36.020347 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.021405 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.023112 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.023984 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.024098 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.024299 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:36.024427 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 14:48:36.024543 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":161,"threads_started":1}
I20260926 14:48:36.025303 18414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.019091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34689 (local address 127.0.0.1:55282)
0926 14:48:36.019809 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:36.019820 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:36.019907 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:36.020472 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:36.020475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:36.020493 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:36.020593 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:36.020599 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.021871 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.021875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:36.022985 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:36.022997 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.024904 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.024908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:36.025001 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.025024 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:48:36.025098 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 14:48:36.025169 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":26,"thread_start_us":91,"threads_started":1}
I20260926 14:48:36.025676 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.018462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43523 (local address 127.0.0.1:58582)
0926 14:48:36.019280 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:36.019293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:36.019387 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:36.019951 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:36.019957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:36.019978 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:36.020156 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:36.020163 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.021374 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.021378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:36.022969 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:36.022980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.025115 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.025119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:36.025207 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.025229 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:48:36.025299 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 14:48:36.025554 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":71,"thread_start_us":99,"threads_started":1}
I20260926 14:48:36.025744 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.019233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.0.98.3:34689)
0926 14:48:36.019683 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:36.019693 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:36.019990 (+ 297us) server_negotiation.cc:407] Connection header received
0926 14:48:36.020064 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:36.020069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:36.020237 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:36.020352 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:36.020731 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.021769 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.023116 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.023984 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.024098 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.024299 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:36.025477 (+ 1178us) server_negotiation.cc:299] Negotiation successful
0926 14:48:36.025592 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":84,"threads_started":1}
I20260926 14:48:36.026473 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.019026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58582 (local address 127.0.98.2:43523)
0926 14:48:36.019456 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:36.019467 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:36.019488 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:48:36.019567 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:36.019573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:36.019784 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:36.020312 (+ 528us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:36.020448 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.021268 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.024370 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.024991 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.025095 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.026144 (+ 1049us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:36.026204 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 14:48:36.026306 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":138,"threads_started":1}
I20260926 14:48:36.028275 18072 tablet_service.cc:1467] Processing CreateTablet for tablet 97caf8419d1e483dba574b46413b925c (DEFAULT_TABLE table=test-table [id=141755ef737947e5bcf14b036ed5e33c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:36.028709 18334 tablet_service.cc:1467] Processing CreateTablet for tablet 97caf8419d1e483dba574b46413b925c (DEFAULT_TABLE table=test-table [id=141755ef737947e5bcf14b036ed5e33c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:36.028927 18203 tablet_service.cc:1467] Processing CreateTablet for tablet 97caf8419d1e483dba574b46413b925c (DEFAULT_TABLE table=test-table [id=141755ef737947e5bcf14b036ed5e33c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:36.029336 18072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97caf8419d1e483dba574b46413b925c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:36.029748 18203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97caf8419d1e483dba574b46413b925c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:36.029748 18334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97caf8419d1e483dba574b46413b925c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:36.035739 18417 tablet_bootstrap.cc:492] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13: Bootstrap starting.
I20260926 14:48:36.037137 18419 tablet_bootstrap.cc:492] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca: Bootstrap starting.
I20260926 14:48:36.038487 18417 tablet_bootstrap.cc:654] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:36.038883 18418 tablet_bootstrap.cc:492] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95: Bootstrap starting.
I20260926 14:48:36.039243 18419 tablet_bootstrap.cc:654] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:36.039362 18417 log.cc:824] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:36.040036 18419 log.cc:824] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:36.040776 18418 tablet_bootstrap.cc:654] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:36.041428 18418 log.cc:824] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:36.042048 18417 tablet_bootstrap.cc:492] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13: No bootstrap required, opened a new log
I20260926 14:48:36.042047 18419 tablet_bootstrap.cc:492] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca: No bootstrap required, opened a new log
I20260926 14:48:36.042245 18419 ts_tablet_manager.cc:1397] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:48:36.042245 18417 ts_tablet_manager.cc:1397] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 14:48:36.043184 18418 tablet_bootstrap.cc:492] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95: No bootstrap required, opened a new log
I20260926 14:48:36.043358 18418 ts_tablet_manager.cc:1397] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:48:36.046416 18418 raft_consensus.cc:348] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056463353d4b4388bcb5e9f1c86f4c13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34699 } } peers { permanent_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "d0056148643040789750a51e7df3bc95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34689 } }
I20260926 14:48:36.046662 18418 raft_consensus.cc:374] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:36.046713 18418 raft_consensus.cc:729] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0056148643040789750a51e7df3bc95, State: Initialized, Role: FOLLOWER
I20260926 14:48:36.046639 18417 raft_consensus.cc:348] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056463353d4b4388bcb5e9f1c86f4c13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34699 } } peers { permanent_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "d0056148643040789750a51e7df3bc95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34689 } }
I20260926 14:48:36.046922 18417 raft_consensus.cc:374] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:36.046983 18417 raft_consensus.cc:729] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056463353d4b4388bcb5e9f1c86f4c13, State: Initialized, Role: FOLLOWER
I20260926 14:48:36.046639 18419 raft_consensus.cc:348] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056463353d4b4388bcb5e9f1c86f4c13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34699 } } peers { permanent_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "d0056148643040789750a51e7df3bc95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34689 } }
I20260926 14:48:36.047304 18419 raft_consensus.cc:374] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:36.047232 18418 consensus_queue.cc:260] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056463353d4b4388bcb5e9f1c86f4c13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34699 } } peers { permanent_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "d0056148643040789750a51e7df3bc95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34689 } }
I20260926 14:48:36.047355 18419 raft_consensus.cc:729] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf211539dc74bd8b6fd4da8c6e133ca, State: Initialized, Role: FOLLOWER
I20260926 14:48:36.047458 18417 consensus_queue.cc:260] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056463353d4b4388bcb5e9f1c86f4c13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34699 } } peers { permanent_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "d0056148643040789750a51e7df3bc95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34689 } }
I20260926 14:48:36.048017 18419 consensus_queue.cc:260] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056463353d4b4388bcb5e9f1c86f4c13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34699 } } peers { permanent_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "d0056148643040789750a51e7df3bc95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34689 } }
I20260926 14:48:36.048239 18418 ts_tablet_manager.cc:1428] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:48:36.048542 18417 ts_tablet_manager.cc:1428] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 14:48:36.048580 18400 heartbeater.cc:503] Master 127.0.98.62:33161 was elected leader, sending a full tablet report...
I20260926 14:48:36.048902 18138 heartbeater.cc:503] Master 127.0.98.62:33161 was elected leader, sending a full tablet report...
I20260926 14:48:36.048951 18419 ts_tablet_manager.cc:1428] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260926 14:48:36.049188 18269 heartbeater.cc:503] Master 127.0.98.62:33161 was elected leader, sending a full tablet report...
I20260926 14:48:36.063730 18424 raft_consensus.cc:482] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:48:36.063967 18424 raft_consensus.cc:504] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056463353d4b4388bcb5e9f1c86f4c13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34699 } } peers { permanent_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "d0056148643040789750a51e7df3bc95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34689 } }
I20260926 14:48:36.065128 18424 leader_election.cc:288] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers baf211539dc74bd8b6fd4da8c6e133ca (127.0.98.2:43523), d0056148643040789750a51e7df3bc95 (127.0.98.3:34689)
I20260926 14:48:36.069823 18416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.065494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38479 (local address 127.0.98.3:34689)
0926 14:48:36.065649 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:36.065654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:36.065669 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:48:36.065841 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:36.065844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:36.065961 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:36.066035 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:36.066311 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.067085 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.068611 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.069219 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.069324 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.069384 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:36.069624 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 14:48:36.069695 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 14:48:36.069823 18415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.065470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39861 (local address 127.0.98.2:43523)
0926 14:48:36.065768 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:36.065772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:36.065939 (+ 167us) server_negotiation.cc:407] Connection header received
0926 14:48:36.066054 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:36.066056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:36.066106 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:36.066170 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:36.066478 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.067209 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.068611 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.069219 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.069324 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.069384 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:36.069624 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 14:48:36.069692 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260926 14:48:36.069845 18140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.065377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34689 (local address 127.0.98.1:38479)
0926 14:48:36.065593 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:36.065610 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:36.065708 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:36.066054 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:36.066057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:36.066070 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:36.066196 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:36.066203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.067184 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.067186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:36.068487 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:36.068499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.069347 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.069351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:36.069485 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.069536 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 14:48:36.069604 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 14:48:36.069683 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":54}
I20260926 14:48:36.069855 18426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.065409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43523 (local address 127.0.98.1:39861)
0926 14:48:36.065855 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:36.065865 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:36.065951 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:36.066212 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:36.066215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:36.066235 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:36.066349 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:36.066355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.067299 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.067303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:36.068487 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:36.068499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.069347 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.069351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:36.069490 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.069526 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 14:48:36.069604 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 14:48:36.069683 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":26,"thread_start_us":126,"threads_started":1}
I20260926 14:48:36.070570 18223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97caf8419d1e483dba574b46413b925c" candidate_uuid: "056463353d4b4388bcb5e9f1c86f4c13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" is_pre_election: true
I20260926 14:48:36.070573 18354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97caf8419d1e483dba574b46413b925c" candidate_uuid: "056463353d4b4388bcb5e9f1c86f4c13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0056148643040789750a51e7df3bc95" is_pre_election: true
I20260926 14:48:36.070880 18223 raft_consensus.cc:2454] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056463353d4b4388bcb5e9f1c86f4c13 in term 0.
I20260926 14:48:36.070876 18354 raft_consensus.cc:2454] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 056463353d4b4388bcb5e9f1c86f4c13 in term 0.
I20260926 14:48:36.071342 18026 leader_election.cc:302] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 056463353d4b4388bcb5e9f1c86f4c13, baf211539dc74bd8b6fd4da8c6e133ca, d0056148643040789750a51e7df3bc95; no voters:
I20260926 14:48:36.071611 18424 raft_consensus.cc:2781] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:48:36.071686 18424 raft_consensus.cc:482] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:48:36.071713 18424 raft_consensus.cc:3037] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:36.072731 18424 raft_consensus.cc:504] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056463353d4b4388bcb5e9f1c86f4c13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34699 } } peers { permanent_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "d0056148643040789750a51e7df3bc95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34689 } }
I20260926 14:48:36.073079 18424 leader_election.cc:288] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [CANDIDATE]: Term 1 election: Requested vote from peers baf211539dc74bd8b6fd4da8c6e133ca (127.0.98.2:43523), d0056148643040789750a51e7df3bc95 (127.0.98.3:34689)
I20260926 14:48:36.073415 18223 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97caf8419d1e483dba574b46413b925c" candidate_uuid: "056463353d4b4388bcb5e9f1c86f4c13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf211539dc74bd8b6fd4da8c6e133ca"
I20260926 14:48:36.073549 18223 raft_consensus.cc:3037] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:36.074482 18223 raft_consensus.cc:2454] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 056463353d4b4388bcb5e9f1c86f4c13 in term 1.
I20260926 14:48:36.074716 18354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97caf8419d1e483dba574b46413b925c" candidate_uuid: "056463353d4b4388bcb5e9f1c86f4c13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0056148643040789750a51e7df3bc95"
I20260926 14:48:36.074797 18028 leader_election.cc:302] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 056463353d4b4388bcb5e9f1c86f4c13, baf211539dc74bd8b6fd4da8c6e133ca; no voters:
I20260926 14:48:36.074849 18354 raft_consensus.cc:3037] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:36.074980 18424 raft_consensus.cc:2781] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:36.075220 18424 raft_consensus.cc:686] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 1 LEADER]: Becoming Leader. State: Replica: 056463353d4b4388bcb5e9f1c86f4c13, State: Running, Role: LEADER
I20260926 14:48:36.075564 18424 consensus_queue.cc:237] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056463353d4b4388bcb5e9f1c86f4c13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34699 } } peers { permanent_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } } peers { permanent_uuid: "d0056148643040789750a51e7df3bc95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34689 } }
I20260926 14:48:36.076040 18354 raft_consensus.cc:2454] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 056463353d4b4388bcb5e9f1c86f4c13 in term 1.
I20260926 14:48:36.078828 17949 catalog_manager.cc:5495] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 056463353d4b4388bcb5e9f1c86f4c13 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "056463353d4b4388bcb5e9f1c86f4c13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "056463353d4b4388bcb5e9f1c86f4c13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34699 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d0056148643040789750a51e7df3bc95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34689 } health_report { overall_health: UNKNOWN } } }
I20260926 14:48:36.103520 18413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:36.099771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.0.98.1:34699)
0926 14:48:36.099920 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:36.099924 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:36.099992 (+ 68us) server_negotiation.cc:407] Connection header received
0926 14:48:36.100080 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:36.100082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:36.100143 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:36.100245 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:48:36.100461 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.101011 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.101916 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:36.102166 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:36.102250 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:36.102546 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:48:36.103089 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:48:36.103188 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:36.103312 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 14:48:36.103369 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 14:48:36.107892 18139 tablet.cc:2369] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:48:36.109758 18223 raft_consensus.cc:1261] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca [term 1 FOLLOWER]: Refusing update from remote peer 056463353d4b4388bcb5e9f1c86f4c13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:36.109758 18354 raft_consensus.cc:1261] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95 [term 1 FOLLOWER]: Refusing update from remote peer 056463353d4b4388bcb5e9f1c86f4c13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:36.110424 18427 consensus_queue.cc:1035] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0056148643040789750a51e7df3bc95" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:48:36.110635 18424 consensus_queue.cc:1035] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf211539dc74bd8b6fd4da8c6e133ca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:48:36.120036 18431 mvcc.cc:205] Tried to move back new op lower bound from 7333618139578159104 to 7333618139448848384. Current Snapshot: MvccSnapshot[applied={T|T < 7333618139578159104}]
I20260926 14:48:36.127192 17950 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:36.127386 17950 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:36.129837 17950 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 702598b796b54bf3bc3620ce3eda2a6f: Sending DeleteTablet for 3 replicas of tablet 97caf8419d1e483dba574b46413b925c
I20260926 14:48:36.130692 18334 tablet_service.cc:1514] Processing DeleteTablet for tablet 97caf8419d1e483dba574b46413b925c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:36 UTC) from {username='slave'} at 127.0.0.1:55282
I20260926 14:48:36.130692 18072 tablet_service.cc:1514] Processing DeleteTablet for tablet 97caf8419d1e483dba574b46413b925c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:36 UTC) from {username='slave'} at 127.0.0.1:35866
I20260926 14:48:36.130735 18203 tablet_service.cc:1514] Processing DeleteTablet for tablet 97caf8419d1e483dba574b46413b925c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:36 UTC) from {username='slave'} at 127.0.0.1:58582
I20260926 14:48:36.131198 18439 tablet_replica.cc:330] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13: stopping tablet replica
I20260926 14:48:36.131378 18440 tablet_replica.cc:330] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95: stopping tablet replica
I20260926 14:48:36.131524 18439 raft_consensus.cc:2229] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:48:36.131871 18441 tablet_replica.cc:330] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca: stopping tablet replica
I20260926 14:48:36.131935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 18010
I20260926 14:48:36.131927 18439 raft_consensus.cc:2258] T 97caf8419d1e483dba574b46413b925c P 056463353d4b4388bcb5e9f1c86f4c13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:36.132584 18441 raft_consensus.cc:2229] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:48:36.132897 18441 raft_consensus.cc:2258] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:36.134147 18440 raft_consensus.cc:2229] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:48:36.134413 18440 raft_consensus.cc:2258] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:36.134687 18441 ts_tablet_manager.cc:1905] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:48:36.135671 18440 ts_tablet_manager.cc:1905] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:48:36.137511 18441 ts_tablet_manager.cc:1918] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:48:36.137609 18441 log.cc:1196] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/97caf8419d1e483dba574b46413b925c
I20260926 14:48:36.137862 18440 ts_tablet_manager.cc:1918] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:48:36.137909 18441 ts_tablet_manager.cc:1939] T 97caf8419d1e483dba574b46413b925c P baf211539dc74bd8b6fd4da8c6e133ca: Deleting consensus metadata
I20260926 14:48:36.137948 18440 log.cc:1196] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/97caf8419d1e483dba574b46413b925c
I20260926 14:48:36.138188 18440 ts_tablet_manager.cc:1939] T 97caf8419d1e483dba574b46413b925c P d0056148643040789750a51e7df3bc95: Deleting consensus metadata
W20260926 14:48:36.139183 17935 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34699 (error 108)
I20260926 14:48:36.139420 17937 catalog_manager.cc:4841] TS baf211539dc74bd8b6fd4da8c6e133ca (127.0.98.2:43523): tablet 97caf8419d1e483dba574b46413b925c (table test-table [id=141755ef737947e5bcf14b036ed5e33c]) successfully deleted
I20260926 14:48:36.139691 17935 catalog_manager.cc:4841] TS d0056148643040789750a51e7df3bc95 (127.0.98.3:34689): tablet 97caf8419d1e483dba574b46413b925c (table test-table [id=141755ef737947e5bcf14b036ed5e33c]) successfully deleted
W20260926 14:48:36.140223 18412 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:48:36.139971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34699 (local address 127.0.0.1:35890)
0926 14:48:36.140081 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:36.140142 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34699: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 14:48:36.140362 17935 catalog_manager.cc:4584] TS 056463353d4b4388bcb5e9f1c86f4c13 (127.0.98.1:34699): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 97caf8419d1e483dba574b46413b925c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34699: connect: Connection reset by peer (error 104)
I20260926 14:48:36.140393 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 18141
I20260926 14:48:36.146127 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 18272
I20260926 14:48:36.151227 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 17919
2026-09-26T14:48:36Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1022 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:36 dist-test-slave-s7k6 krb5kdc[18445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:36 dist-test-slave-s7k6 krb5kdc[18445](info): set up 2 sockets
Sep 26 14:48:36 dist-test-slave-s7k6 krb5kdc[18445](info): commencing operation
krb5kdc: starting...
W20260926 14:48:37.126602 392 mini_kdc.cc:121] Time spent starting KDC: real 0.955s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:37 dist-test-slave-s7k6 krb5kdc[18445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:48:37.246228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43959
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:37.360678 18458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:37.360981 18458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:37.361037 18458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:37.365293 18458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:37.365372 18458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:37.365401 18458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:37.365422 18458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:37.365449 18458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:37.371172 18458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:37.372373 18458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:37.373440 18458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:37.382241 18463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:37.382241 18466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:37.382452 18458 server_base.cc:1034] running on GCE node
W20260926 14:48:37.382706 18464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:37.383332 18458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:37.384553 18458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:37.385749 18458 hybrid_clock.cc:648] HybridClock initialized: now 1790434117385731 us; error 37 us; skew 500 ppm
Sep 26 14:48:37 dist-test-slave-s7k6 krb5kdc[18445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:37.388715 18458 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:48:37.389894 18458 webserver.cc:460] Webserver started at http://127.0.98.62:43203/ using document root <none> and password file <none>
I20260926 14:48:37.390604 18458 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:37.390681 18458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:37.390933 18458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:37.393150 18458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "9cda9d37f8184dd6a78cf2c609ffe00f"
format_stamp: "Formatted at 2026-09-26 14:48:37 on dist-test-slave-s7k6"
server_key: "ada2ef994c5ba2c9f2511db74e634c21"
server_key_iv: "d11b2ed8220960603495244071014e28"
server_key_version: "encryptionkey@0"
I20260926 14:48:37.393734 18458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "9cda9d37f8184dd6a78cf2c609ffe00f"
format_stamp: "Formatted at 2026-09-26 14:48:37 on dist-test-slave-s7k6"
server_key: "ada2ef994c5ba2c9f2511db74e634c21"
server_key_iv: "d11b2ed8220960603495244071014e28"
server_key_version: "encryptionkey@0"
I20260926 14:48:37.397720 18458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:48:37.400338 18473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:37.401659 18458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:48:37.401784 18458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "9cda9d37f8184dd6a78cf2c609ffe00f"
format_stamp: "Formatted at 2026-09-26 14:48:37 on dist-test-slave-s7k6"
server_key: "ada2ef994c5ba2c9f2511db74e634c21"
server_key_iv: "d11b2ed8220960603495244071014e28"
server_key_version: "encryptionkey@0"
I20260926 14:48:37.401921 18458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:37.420514 18458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:37.423920 18458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:37.424116 18458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:37.431336 18458 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43959
I20260926 14:48:37.431346 18525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43959 every 8 connection(s)
I20260926 14:48:37.432408 18458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:37.435316 18526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:37.441075 18526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f: Bootstrap starting.
I20260926 14:48:37.442602 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 18458
I20260926 14:48:37.442734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:37.443041 392 external_mini_cluster.cc:1428] Setting key 8788c5b3667188e3d87b379d6449660b
I20260926 14:48:37.443727 18526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:37.444641 18526 log.cc:824] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:37.446583 18526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f: No bootstrap required, opened a new log
I20260926 14:48:37.450047 18526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cda9d37f8184dd6a78cf2c609ffe00f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } }
I20260926 14:48:37.450299 18526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:37.450356 18526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cda9d37f8184dd6a78cf2c609ffe00f, State: Initialized, Role: FOLLOWER
I20260926 14:48:37.450824 18526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cda9d37f8184dd6a78cf2c609ffe00f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } }
I20260926 14:48:37.450961 18526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:37.451030 18526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:37.451128 18526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [term 0 FOLLOWER]: Advancing to term 1
Sep 26 14:48:37 dist-test-slave-s7k6 krb5kdc[18445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:48:37.452090 18526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cda9d37f8184dd6a78cf2c609ffe00f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } }
I20260926 14:48:37.452512 18526 leader_election.cc:302] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cda9d37f8184dd6a78cf2c609ffe00f; no voters:
I20260926 14:48:37.452881 18526 leader_election.cc:288] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:37.452970 18531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:37.453141 18531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [term 1 LEADER]: Becoming Leader. State: Replica: 9cda9d37f8184dd6a78cf2c609ffe00f, State: Running, Role: LEADER
I20260926 14:48:37.453467 18531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cda9d37f8184dd6a78cf2c609ffe00f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } }
I20260926 14:48:37.454247 18526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:37.455201 18533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cda9d37f8184dd6a78cf2c609ffe00f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cda9d37f8184dd6a78cf2c609ffe00f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } } }
I20260926 14:48:37.455353 18533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:37.455752 18540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:37.455770 18532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cda9d37f8184dd6a78cf2c609ffe00f. Latest consensus state: current_term: 1 leader_uuid: "9cda9d37f8184dd6a78cf2c609ffe00f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cda9d37f8184dd6a78cf2c609ffe00f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43959 } } }
I20260926 14:48:37.455978 18532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:37.459271 18540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:37.459975 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:37.444163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.0.98.62:43959)
0926 14:48:37.444619 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:37.444634 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:37.444669 (+ 35us) server_negotiation.cc:407] Connection header received
0926 14:48:37.445496 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:37.445518 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:37.445865 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:37.446203 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:37.447027 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:37.448054 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:37.448930 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:37.449435 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:37.449591 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:37.452082 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:37.452088 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:37.452102 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:37.452132 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:37.456092 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:37.456683 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:37.456689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:37.456694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:37.456858 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:37.457207 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:37.457210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:37.457212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:37.457680 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:37.458464 (+ 784us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:37.458598 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 14:48:37.458932 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":152,"threads_started":1}
I20260926 14:48:37.464895 18540 catalog_manager.cc:1347] Generated new cluster ID: 7857fc2f1d154d6f8ab37910ce43acbf
I20260926 14:48:37.464967 18540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:37.478338 18540 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:37.479631 18540 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:37.486825 18540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f: Generated new TSK 0
I20260926 14:48:37.487579 18540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:48:37.523907 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43959
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:37.664970 18552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:37.665259 18552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:37.665325 18552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:37.669514 18552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:37.669587 18552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:37.669670 18552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:48:37.675410 18552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:37.676468 18552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:37.677500 18552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:37.686888 18558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:37.686832 18560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:37.686910 18557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:37.687143 18552 server_base.cc:1034] running on GCE node
I20260926 14:48:37.687711 18552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:37.688359 18552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:37.689538 18552 hybrid_clock.cc:648] HybridClock initialized: now 1790434117689494 us; error 59 us; skew 500 ppm
Sep 26 14:48:37 dist-test-slave-s7k6 krb5kdc[18445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:37.692322 18552 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:48:37.693217 18552 webserver.cc:460] Webserver started at http://127.0.98.1:41021/ using document root <none> and password file <none>
I20260926 14:48:37.693765 18552 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:37.693831 18552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:37.694031 18552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:37.696102 18552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "b7248a99679b479484b3cb9edb249899"
format_stamp: "Formatted at 2026-09-26 14:48:37 on dist-test-slave-s7k6"
server_key: "77ad61fe9830e3d741343d904978d218"
server_key_iv: "f09a827ce9516ed7b40714648c93eed3"
server_key_version: "encryptionkey@0"
I20260926 14:48:37.696673 18552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b7248a99679b479484b3cb9edb249899"
format_stamp: "Formatted at 2026-09-26 14:48:37 on dist-test-slave-s7k6"
server_key: "77ad61fe9830e3d741343d904978d218"
server_key_iv: "f09a827ce9516ed7b40714648c93eed3"
server_key_version: "encryptionkey@0"
I20260926 14:48:37.700294 18552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 14:48:37.702934 18567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:37.704206 18552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:48:37.704320 18552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "b7248a99679b479484b3cb9edb249899"
format_stamp: "Formatted at 2026-09-26 14:48:37 on dist-test-slave-s7k6"
server_key: "77ad61fe9830e3d741343d904978d218"
server_key_iv: "f09a827ce9516ed7b40714648c93eed3"
server_key_version: "encryptionkey@0"
I20260926 14:48:37.704432 18552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:37.743854 18552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:37.747403 18552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:37.747597 18552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:37.748242 18552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:37.749135 18552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:37.749182 18552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:37.749228 18552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:37.749255 18552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:37.758037 18552 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40253
I20260926 14:48:37.758064 18680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40253 every 8 connection(s)
I20260926 14:48:37.759006 18552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:48:37.761493 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 18552
I20260926 14:48:37.761620 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:48:37.761910 392 external_mini_cluster.cc:1428] Setting key 5d874bd4b21ac9fd6b1e17ba6352f832
Sep 26 14:48:37 dist-test-slave-s7k6 krb5kdc[18445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 14:48:37.773861 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:37.761419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38123 (local address 127.0.98.62:43959)
0926 14:48:37.761567 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:37.761572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:37.762563 (+ 991us) server_negotiation.cc:407] Connection header received
0926 14:48:37.763697 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:37.763701 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:37.763757 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:37.763850 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:37.765074 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:37.765869 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:37.767116 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:37.767387 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:37.767485 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:37.770282 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:37.770285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:37.770287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:37.770312 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:37.771801 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:37.772327 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:37.772331 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:37.772333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:37.772415 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:37.772749 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:37.772752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:37.772754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:37.772945 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:37.773031 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:37.773562 (+ 531us) server_negotiation.cc:299] Negotiation successful
0926 14:48:37.773698 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:48:37.774781 18683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:37.761840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43959 (local address 127.0.98.1:38123)
0926 14:48:37.762377 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:37.762444 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:37.763484 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:37.763995 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:37.764002 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:37.764373 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:37.764897 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:37.764913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:37.765991 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:37.765993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:37.766990 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:37.767001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:37.767546 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:37.767551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:37.767728 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:37.768377 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:48:37.768394 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:37.770070 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:37.771918 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:37.771924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:37.771942 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:37.772225 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:37.772500 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:37.772503 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:37.772505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:37.772653 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:37.773041 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:37.773048 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:37.773315 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 14:48:37.773507 (+ 192us) client_negotiation.cc:240] Negotiation successful
0926 14:48:37.773858 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":150,"threads_started":1}
I20260926 14:48:37.776137 18681 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43959
I20260926 14:48:37.776384 18681 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:37.776891 18681 heartbeater.cc:511] Master 127.0.98.62:43959 requested a full tablet report, sending...
I20260926 14:48:37.778482 18490 ts_manager.cc:194] Registered new tserver with Master: b7248a99679b479484b3cb9edb249899 (127.0.98.1:40253)
I20260926 14:48:37.779697 18490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:48:37.787627 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43959
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:37.900015 18686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:37.900297 18686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:37.900359 18686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:37.904623 18686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:37.904697 18686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:37.904781 18686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:48:37.910583 18686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:37.911795 18686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:37.912910 18686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:37.923465 18694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:37.923470 18692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:37.923457 18691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:37.923663 18686 server_base.cc:1034] running on GCE node
I20260926 14:48:37.924216 18686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:37.924926 18686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:37.926129 18686 hybrid_clock.cc:648] HybridClock initialized: now 1790434117926105 us; error 37 us; skew 500 ppm
Sep 26 14:48:37 dist-test-slave-s7k6 krb5kdc[18445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:37.935622 18686 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:48:37.936622 18686 webserver.cc:460] Webserver started at http://127.0.98.2:45705/ using document root <none> and password file <none>
I20260926 14:48:37.937186 18686 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:37.937247 18686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:37.937419 18686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:37.939630 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "f8004bbfaf3447d48be325dad50098ef"
format_stamp: "Formatted at 2026-09-26 14:48:37 on dist-test-slave-s7k6"
server_key: "aefbcc454544265cad330b96219180a1"
server_key_iv: "49c7329967c650416d170f2ee9aae829"
server_key_version: "encryptionkey@0"
I20260926 14:48:37.940182 18686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f8004bbfaf3447d48be325dad50098ef"
format_stamp: "Formatted at 2026-09-26 14:48:37 on dist-test-slave-s7k6"
server_key: "aefbcc454544265cad330b96219180a1"
server_key_iv: "49c7329967c650416d170f2ee9aae829"
server_key_version: "encryptionkey@0"
I20260926 14:48:37.944131 18686 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:48:37.946653 18701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:37.948155 18686 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:37.948276 18686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "f8004bbfaf3447d48be325dad50098ef"
format_stamp: "Formatted at 2026-09-26 14:48:37 on dist-test-slave-s7k6"
server_key: "aefbcc454544265cad330b96219180a1"
server_key_iv: "49c7329967c650416d170f2ee9aae829"
server_key_version: "encryptionkey@0"
I20260926 14:48:37.948402 18686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:37.978006 18686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:37.981801 18686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:37.981976 18686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:37.982630 18686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:37.983560 18686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:37.983611 18686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:37.983655 18686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:37.983685 18686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:37.992532 18686 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36879
I20260926 14:48:37.992552 18814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36879 every 8 connection(s)
I20260926 14:48:37.993548 18686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:48:37.994095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 18686
I20260926 14:48:37.994226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:48:37.994532 392 external_mini_cluster.cc:1428] Setting key 84d1e66f6f6e0c76871921bc0bbbaa8b
Sep 26 14:48:37 dist-test-slave-s7k6 krb5kdc[18445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:48:38.008548 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:37.995418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36245 (local address 127.0.98.62:43959)
0926 14:48:37.995529 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:37.995532 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:37.996709 (+ 1177us) server_negotiation.cc:407] Connection header received
0926 14:48:37.997827 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:37.997832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:37.997896 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:37.997989 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:37.999343 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:37.999882 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.001635 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.001870 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.001958 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.004760 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:38.004763 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:38.004766 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:38.004791 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:38.006131 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:38.006675 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:38.006679 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:38.006681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:38.006802 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:38.007243 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:38.007245 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:38.007247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:38.007482 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:38.007572 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:38.008244 (+ 672us) server_negotiation.cc:299] Negotiation successful
0926 14:48:38.008377 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:48:38.009424 18817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:37.995981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43959 (local address 127.0.98.2:36245)
0926 14:48:37.996532 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:37.996593 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:37.997580 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:37.998158 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:37.998165 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:37.998526 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:37.999172 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:37.999194 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.000015 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.000019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.001484 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.001499 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.002029 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.002034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.002258 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.002922 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:48:38.002945 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:38.004582 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:38.006226 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:38.006232 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:38.006245 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:38.006575 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:38.006891 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:38.006895 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:38.006897 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:38.007147 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:38.007578 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:38.007585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:38.007917 (+ 332us) client_negotiation.cc:769] Sending connection context
0926 14:48:38.008150 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 14:48:38.008532 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":134,"threads_started":1}
I20260926 14:48:38.010699 18815 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43959
I20260926 14:48:38.010989 18815 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:38.011613 18815 heartbeater.cc:511] Master 127.0.98.62:43959 requested a full tablet report, sending...
I20260926 14:48:38.012849 18490 ts_manager.cc:194] Registered new tserver with Master: f8004bbfaf3447d48be325dad50098ef (127.0.98.2:36879)
I20260926 14:48:38.013624 18490 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:48:38.021818 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43959
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:38.135030 18820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:38.135311 18820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:38.135365 18820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:38.139660 18820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:38.139735 18820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:38.139796 18820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:48:38.145401 18820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:38.146451 18820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:38.147588 18820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:38.157373 18828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:38.157373 18826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:38.157410 18820 server_base.cc:1034] running on GCE node
W20260926 14:48:38.157367 18825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:38.157997 18820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:38.158674 18820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:38.159837 18820 hybrid_clock.cc:648] HybridClock initialized: now 1790434118159797 us; error 56 us; skew 500 ppm
Sep 26 14:48:38 dist-test-slave-s7k6 krb5kdc[18445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:38.162694 18820 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:48:38.163683 18820 webserver.cc:460] Webserver started at http://127.0.98.3:44287/ using document root <none> and password file <none>
I20260926 14:48:38.164274 18820 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:38.164346 18820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:38.164547 18820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:38.166824 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "a41ccac654b7422eb5427e6014b48898"
format_stamp: "Formatted at 2026-09-26 14:48:38 on dist-test-slave-s7k6"
server_key: "bd7fadcbe7ef138aef80b28f07d978bd"
server_key_iv: "f80b278c2d9e640562787c9056da03dc"
server_key_version: "encryptionkey@0"
I20260926 14:48:38.167424 18820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a41ccac654b7422eb5427e6014b48898"
format_stamp: "Formatted at 2026-09-26 14:48:38 on dist-test-slave-s7k6"
server_key: "bd7fadcbe7ef138aef80b28f07d978bd"
server_key_iv: "f80b278c2d9e640562787c9056da03dc"
server_key_version: "encryptionkey@0"
I20260926 14:48:38.171365 18820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260926 14:48:38.174105 18835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:38.175545 18820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:38.175683 18820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "a41ccac654b7422eb5427e6014b48898"
format_stamp: "Formatted at 2026-09-26 14:48:38 on dist-test-slave-s7k6"
server_key: "bd7fadcbe7ef138aef80b28f07d978bd"
server_key_iv: "f80b278c2d9e640562787c9056da03dc"
server_key_version: "encryptionkey@0"
I20260926 14:48:38.175796 18820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:38.198370 18820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:38.201819 18820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:38.202018 18820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:38.202677 18820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:38.203581 18820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:38.203630 18820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:38.203686 18820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:38.203713 18820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:38.212568 18820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42301
I20260926 14:48:38.212592 18948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42301 every 8 connection(s)
I20260926 14:48:38.213502 18820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:48:38.218133 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 18820
I20260926 14:48:38.218263 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:48:38.218565 392 external_mini_cluster.cc:1428] Setting key 975587e1cdc539a0c5aa98a52df35297
Sep 26 14:48:38 dist-test-slave-s7k6 krb5kdc[18445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:48:38.227648 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.215482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35575 (local address 127.0.98.62:43959)
0926 14:48:38.215629 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:38.215634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:38.216664 (+ 1030us) server_negotiation.cc:407] Connection header received
0926 14:48:38.217662 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:38.217666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:38.217719 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:38.217796 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:38.219376 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.219916 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.221156 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.221395 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.221486 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.224262 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:38.224265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:38.224267 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:38.224295 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:38.225559 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:38.226075 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:38.226078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:38.226079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:38.226165 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:38.226488 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:38.226492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:38.226493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:38.226688 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:38.226823 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:38.227346 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 14:48:38.227481 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:48:38.228504 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.215960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43959 (local address 127.0.98.3:35575)
0926 14:48:38.216501 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:38.216547 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:38.217427 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:38.218055 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:38.218062 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:38.218526 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:38.219210 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.219233 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.220047 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.220051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.221018 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.221029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.221565 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.221570 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.221744 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.222409 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:48:38.222427 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:38.224080 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:38.225678 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:38.225684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:38.225702 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:38.225974 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:38.226249 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:38.226252 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:38.226254 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:38.226398 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:38.226811 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:38.226818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:38.227095 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 14:48:38.227274 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 14:48:38.227631 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":136,"threads_started":1}
I20260926 14:48:38.229699 18949 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43959
I20260926 14:48:38.229928 18949 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:38.230481 18949 heartbeater.cc:511] Master 127.0.98.62:43959 requested a full tablet report, sending...
I20260926 14:48:38.231694 18486 ts_manager.cc:194] Registered new tserver with Master: a41ccac654b7422eb5427e6014b48898 (127.0.98.3:42301)
I20260926 14:48:38.232393 18486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35575
I20260926 14:48:38.232699 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:48:38.242063 18529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.234951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.0.98.62:43959)
0926 14:48:38.235070 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:38.235074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:38.235266 (+ 192us) server_negotiation.cc:407] Connection header received
0926 14:48:38.235364 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:38.235366 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:38.235404 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:38.235470 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:38.236128 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.236638 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.237518 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.237741 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.237871 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.239311 (+ 1440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:38.239315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:38.239317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:38.239340 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:38.240458 (+ 1118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:38.240974 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:38.240977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:38.240979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:38.241069 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:38.241308 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:38.241310 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:38.241311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:38.241499 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:38.241615 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:38.241860 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 14:48:38.241945 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:48:38.245036 18486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:48:38.248127 18486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:48:38.263446 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.256986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40253 (local address 127.0.0.1:41388)
0926 14:48:38.258051 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:38.258062 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:38.258143 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:38.258579 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:38.258582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:38.258598 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:38.258678 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.258684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.260015 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.260019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.261452 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.261464 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.262685 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.262690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.262937 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.263009 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 14:48:38.263134 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 14:48:38.263277 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":308,"thread_start_us":89,"threads_started":1}
I20260926 14:48:38.263466 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.257389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.0.98.1:40253)
0926 14:48:38.257849 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:38.257868 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:38.258167 (+ 299us) server_negotiation.cc:407] Connection header received
0926 14:48:38.258248 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:38.258263 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:38.258456 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:38.258608 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:38.258780 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.259872 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.261593 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.262547 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.262667 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.262866 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:38.263194 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 14:48:38.263301 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":121,"threads_started":1}
I20260926 14:48:38.264318 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.256764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42301 (local address 127.0.0.1:50010)
0926 14:48:38.257993 (+ 1229us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:38.258007 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:38.258088 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:38.258801 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:38.258804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:38.258819 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:38.258910 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.258927 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.259858 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.259862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.261606 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.261618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.263845 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.263850 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.263984 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.264012 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 14:48:38.264088 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 14:48:38.264170 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1168,"mutex_wait_us":290,"thread_start_us":89,"threads_started":1}
I20260926 14:48:38.264838 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.257397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.0.98.3:42301)
0926 14:48:38.258361 (+ 964us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:38.258373 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:38.258393 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:48:38.258473 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:38.258478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:38.258666 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:38.258828 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:38.259006 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.259738 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.261736 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.262547 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.262667 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.262867 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:38.264551 (+ 1684us) server_negotiation.cc:299] Negotiation successful
0926 14:48:38.264673 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":869,"thread_start_us":114,"threads_started":1}
I20260926 14:48:38.267584 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.256587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36879 (local address 127.0.0.1:37036)
0926 14:48:38.257279 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:38.257335 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:38.257466 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:38.258244 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:38.258250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:38.258271 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:38.258405 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.258412 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.259933 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.259936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.261606 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.261618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.267146 (+ 5528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.267150 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.267254 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.267280 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 14:48:38.267349 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 14:48:38.267442 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":342,"thread_start_us":78,"threads_started":1}
I20260926 14:48:38.268175 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.256772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.0.98.2:36879)
0926 14:48:38.257807 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:38.257817 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:38.257837 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:48:38.257900 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:38.257904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:38.258071 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:38.258254 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:38.258535 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.259808 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.265855 (+ 6047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.266743 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.266896 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.267071 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:38.267890 (+ 819us) server_negotiation.cc:299] Negotiation successful
0926 14:48:38.268019 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":97,"threads_started":1}
I20260926 14:48:38.267803 18615 tablet_service.cc:1467] Processing CreateTablet for tablet bd521a8d4573405a9e815b43057197aa (DEFAULT_TABLE table=test-table [id=b3ee37a5990c402b9fdb8985e0a81c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:38.269111 18615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd521a8d4573405a9e815b43057197aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:38.270005 18883 tablet_service.cc:1467] Processing CreateTablet for tablet bd521a8d4573405a9e815b43057197aa (DEFAULT_TABLE table=test-table [id=b3ee37a5990c402b9fdb8985e0a81c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:38.271209 18883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd521a8d4573405a9e815b43057197aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:38.271004 18749 tablet_service.cc:1467] Processing CreateTablet for tablet bd521a8d4573405a9e815b43057197aa (DEFAULT_TABLE table=test-table [id=b3ee37a5990c402b9fdb8985e0a81c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:38.271979 18749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd521a8d4573405a9e815b43057197aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:38.277143 18966 tablet_bootstrap.cc:492] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899: Bootstrap starting.
I20260926 14:48:38.279493 18968 tablet_bootstrap.cc:492] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: Bootstrap starting.
I20260926 14:48:38.279556 18966 tablet_bootstrap.cc:654] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:38.280498 18966 log.cc:824] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:38.281195 18967 tablet_bootstrap.cc:492] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef: Bootstrap starting.
I20260926 14:48:38.281749 18968 tablet_bootstrap.cc:654] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:38.282706 18966 tablet_bootstrap.cc:492] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899: No bootstrap required, opened a new log
I20260926 14:48:38.282954 18966 ts_tablet_manager.cc:1397] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20260926 14:48:38.282995 18968 log.cc:824] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:38.283449 18967 tablet_bootstrap.cc:654] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:38.284262 18967 log.cc:824] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:38.285164 18968 tablet_bootstrap.cc:492] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: No bootstrap required, opened a new log
I20260926 14:48:38.285386 18968 ts_tablet_manager.cc:1397] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 14:48:38.285913 18967 tablet_bootstrap.cc:492] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef: No bootstrap required, opened a new log
I20260926 14:48:38.286082 18967 ts_tablet_manager.cc:1397] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:48:38.287407 18966 raft_consensus.cc:348] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8004bbfaf3447d48be325dad50098ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "a41ccac654b7422eb5427e6014b48898" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42301 } } peers { permanent_uuid: "b7248a99679b479484b3cb9edb249899" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40253 } }
I20260926 14:48:38.287693 18966 raft_consensus.cc:374] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:38.287766 18966 raft_consensus.cc:729] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7248a99679b479484b3cb9edb249899, State: Initialized, Role: FOLLOWER
I20260926 14:48:38.288242 18966 consensus_queue.cc:260] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8004bbfaf3447d48be325dad50098ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "a41ccac654b7422eb5427e6014b48898" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42301 } } peers { permanent_uuid: "b7248a99679b479484b3cb9edb249899" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40253 } }
I20260926 14:48:38.288944 18681 heartbeater.cc:503] Master 127.0.98.62:43959 was elected leader, sending a full tablet report...
I20260926 14:48:38.289206 18968 raft_consensus.cc:348] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8004bbfaf3447d48be325dad50098ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "a41ccac654b7422eb5427e6014b48898" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42301 } } peers { permanent_uuid: "b7248a99679b479484b3cb9edb249899" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40253 } }
I20260926 14:48:38.289291 18967 raft_consensus.cc:348] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8004bbfaf3447d48be325dad50098ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "a41ccac654b7422eb5427e6014b48898" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42301 } } peers { permanent_uuid: "b7248a99679b479484b3cb9edb249899" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40253 } }
I20260926 14:48:38.289490 18968 raft_consensus.cc:374] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:38.289518 18967 raft_consensus.cc:374] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:38.289546 18968 raft_consensus.cc:729] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a41ccac654b7422eb5427e6014b48898, State: Initialized, Role: FOLLOWER
I20260926 14:48:38.289562 18967 raft_consensus.cc:729] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8004bbfaf3447d48be325dad50098ef, State: Initialized, Role: FOLLOWER
I20260926 14:48:38.289969 18968 consensus_queue.cc:260] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8004bbfaf3447d48be325dad50098ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "a41ccac654b7422eb5427e6014b48898" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42301 } } peers { permanent_uuid: "b7248a99679b479484b3cb9edb249899" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40253 } }
I20260926 14:48:38.289979 18967 consensus_queue.cc:260] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8004bbfaf3447d48be325dad50098ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "a41ccac654b7422eb5427e6014b48898" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42301 } } peers { permanent_uuid: "b7248a99679b479484b3cb9edb249899" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40253 } }
I20260926 14:48:38.290441 18966 ts_tablet_manager.cc:1428] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260926 14:48:38.290940 18967 ts_tablet_manager.cc:1428] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:48:38.290952 18968 ts_tablet_manager.cc:1428] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260926 14:48:38.291164 18949 heartbeater.cc:503] Master 127.0.98.62:43959 was elected leader, sending a full tablet report...
I20260926 14:48:38.291240 18815 heartbeater.cc:503] Master 127.0.98.62:43959 was elected leader, sending a full tablet report...
I20260926 14:48:38.415118 18974 raft_consensus.cc:482] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:48:38.415352 18974 raft_consensus.cc:504] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8004bbfaf3447d48be325dad50098ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "a41ccac654b7422eb5427e6014b48898" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42301 } } peers { permanent_uuid: "b7248a99679b479484b3cb9edb249899" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40253 } }
I20260926 14:48:38.416522 18974 leader_election.cc:288] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8004bbfaf3447d48be325dad50098ef (127.0.98.2:36879), b7248a99679b479484b3cb9edb249899 (127.0.98.1:40253)
I20260926 14:48:38.421605 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.416865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38453 (local address 127.0.98.2:36879)
0926 14:48:38.416993 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:38.416998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:38.417015 (+ 17us) server_negotiation.cc:407] Connection header received
0926 14:48:38.417153 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:38.417156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:38.417222 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:38.417290 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:38.417619 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.418538 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.420227 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.420871 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.420994 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.421063 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:38.421376 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 14:48:38.421459 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 14:48:38.421605 18965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.416937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37277 (local address 127.0.98.1:40253)
0926 14:48:38.417055 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:38.417059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:38.417352 (+ 293us) server_negotiation.cc:407] Connection header received
0926 14:48:38.417484 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:38.417486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:38.417529 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:38.417598 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:38.417980 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.418719 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.420227 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.420871 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.420994 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.421064 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:38.421376 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 14:48:38.421459 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:48:38.421630 18951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.416784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36879 (local address 127.0.98.3:38453)
0926 14:48:38.416908 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:38.416923 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:38.417018 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:38.417333 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:38.417336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:38.417378 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:38.417507 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.417513 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.418669 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.418671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.420047 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.420062 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.421042 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.421046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.421223 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.421258 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 14:48:38.421343 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:48:38.421460 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":26,"spinlock_wait_cycles":32256}
I20260926 14:48:38.421638 18975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.416793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40253 (local address 127.0.98.3:37277)
0926 14:48:38.417248 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:38.417259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:38.417385 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:38.417621 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:38.417623 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:38.417640 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:38.417815 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.417821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.418837 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.418840 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.420049 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.420062 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.421042 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.421046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.421240 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.421263 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:48:38.421343 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 14:48:38.421464 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":233,"spinlock_wait_cycles":8960,"thread_start_us":145,"threads_started":1}
I20260926 14:48:38.422492 18769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd521a8d4573405a9e815b43057197aa" candidate_uuid: "a41ccac654b7422eb5427e6014b48898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8004bbfaf3447d48be325dad50098ef" is_pre_election: true
I20260926 14:48:38.422731 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd521a8d4573405a9e815b43057197aa" candidate_uuid: "a41ccac654b7422eb5427e6014b48898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7248a99679b479484b3cb9edb249899" is_pre_election: true
I20260926 14:48:38.422868 18769 raft_consensus.cc:2454] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41ccac654b7422eb5427e6014b48898 in term 0.
I20260926 14:48:38.423060 18635 raft_consensus.cc:2454] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a41ccac654b7422eb5427e6014b48898 in term 0.
I20260926 14:48:38.423381 18839 leader_election.cc:302] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41ccac654b7422eb5427e6014b48898, f8004bbfaf3447d48be325dad50098ef; no voters:
I20260926 14:48:38.423684 18974 raft_consensus.cc:2781] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:48:38.423776 18974 raft_consensus.cc:482] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:48:38.423833 18974 raft_consensus.cc:3037] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:38.424863 18974 raft_consensus.cc:504] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8004bbfaf3447d48be325dad50098ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "a41ccac654b7422eb5427e6014b48898" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42301 } } peers { permanent_uuid: "b7248a99679b479484b3cb9edb249899" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40253 } }
I20260926 14:48:38.425189 18974 leader_election.cc:288] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [CANDIDATE]: Term 1 election: Requested vote from peers f8004bbfaf3447d48be325dad50098ef (127.0.98.2:36879), b7248a99679b479484b3cb9edb249899 (127.0.98.1:40253)
I20260926 14:48:38.425581 18769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd521a8d4573405a9e815b43057197aa" candidate_uuid: "a41ccac654b7422eb5427e6014b48898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8004bbfaf3447d48be325dad50098ef"
I20260926 14:48:38.425592 18635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd521a8d4573405a9e815b43057197aa" candidate_uuid: "a41ccac654b7422eb5427e6014b48898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7248a99679b479484b3cb9edb249899"
I20260926 14:48:38.425699 18635 raft_consensus.cc:3037] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:38.425710 18769 raft_consensus.cc:3037] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:38.426796 18635 raft_consensus.cc:2454] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41ccac654b7422eb5427e6014b48898 in term 1.
I20260926 14:48:38.426810 18769 raft_consensus.cc:2454] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a41ccac654b7422eb5427e6014b48898 in term 1.
I20260926 14:48:38.427153 18838 leader_election.cc:302] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a41ccac654b7422eb5427e6014b48898, b7248a99679b479484b3cb9edb249899; no voters:
I20260926 14:48:38.427322 18974 raft_consensus.cc:2781] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:38.427532 18974 raft_consensus.cc:686] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 1 LEADER]: Becoming Leader. State: Replica: a41ccac654b7422eb5427e6014b48898, State: Running, Role: LEADER
I20260926 14:48:38.427907 18974 consensus_queue.cc:237] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8004bbfaf3447d48be325dad50098ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } } peers { permanent_uuid: "a41ccac654b7422eb5427e6014b48898" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42301 } } peers { permanent_uuid: "b7248a99679b479484b3cb9edb249899" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40253 } }
I20260926 14:48:38.431142 18486 catalog_manager.cc:5495] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 reported cstate change: term changed from 0 to 1, leader changed from <none> to a41ccac654b7422eb5427e6014b48898 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "a41ccac654b7422eb5427e6014b48898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8004bbfaf3447d48be325dad50098ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a41ccac654b7422eb5427e6014b48898" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42301 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7248a99679b479484b3cb9edb249899" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40253 } health_report { overall_health: UNKNOWN } } }
I20260926 14:48:38.453037 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.449559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.0.98.3:42301)
0926 14:48:38.449693 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:38.449697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:38.449711 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:48:38.449806 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:38.449808 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:38.449872 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:38.449961 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:48:38.450209 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.450703 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.451548 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.451777 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.451863 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.452200 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:48:38.452626 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:48:38.452755 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:38.452871 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 14:48:38.452909 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 14:48:38.459143 18635 raft_consensus.cc:1261] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899 [term 1 FOLLOWER]: Refusing update from remote peer a41ccac654b7422eb5427e6014b48898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:38.459143 18769 raft_consensus.cc:1261] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef [term 1 FOLLOWER]: Refusing update from remote peer a41ccac654b7422eb5427e6014b48898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:38.459919 18976 consensus_queue.cc:1035] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7248a99679b479484b3cb9edb249899" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:48:38.460145 18974 consensus_queue.cc:1035] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8004bbfaf3447d48be325dad50098ef" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 14:48:38.464787 18950 tablet.cc:2369] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:48:38.468225 18981 mvcc.cc:205] Tried to move back new op lower bound from 7333618149201944576 to 7333618149083865088. Current Snapshot: MvccSnapshot[applied={T|T < 7333618149201944576}]
I20260926 14:48:38.477214 18490 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:38.477367 18490 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:38.479664 18490 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9cda9d37f8184dd6a78cf2c609ffe00f: Sending DeleteTablet for 3 replicas of tablet bd521a8d4573405a9e815b43057197aa
I20260926 14:48:38.480414 18883 tablet_service.cc:1514] Processing DeleteTablet for tablet bd521a8d4573405a9e815b43057197aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50010
I20260926 14:48:38.480400 18749 tablet_service.cc:1514] Processing DeleteTablet for tablet bd521a8d4573405a9e815b43057197aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37036
I20260926 14:48:38.480566 18615 tablet_service.cc:1514] Processing DeleteTablet for tablet bd521a8d4573405a9e815b43057197aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41388
I20260926 14:48:38.481070 18991 tablet_replica.cc:330] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899: stopping tablet replica
I20260926 14:48:38.481076 18989 tablet_replica.cc:330] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: stopping tablet replica
I20260926 14:48:38.481452 18989 raft_consensus.cc:2229] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:48:38.481489 18991 raft_consensus.cc:2229] T bd521a8d4573405a9e815b43057197aa P b7248a99679b479484b3cb9edb249899 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:48:38.481637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 18552
I20260926 14:48:38.482681 18989 raft_consensus.cc:2258] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:38.484813 18989 ts_tablet_manager.cc:1905] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:48:38.487018 18990 tablet_replica.cc:330] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef: stopping tablet replica
W20260926 14:48:38.487516 18476 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40253 (error 108)
I20260926 14:48:38.487676 18989 ts_tablet_manager.cc:1918] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:48:38.487776 18989 log.cc:1196] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/bd521a8d4573405a9e815b43057197aa
I20260926 14:48:38.487944 18990 raft_consensus.cc:2229] T bd521a8d4573405a9e815b43057197aa P f8004bbfaf3447d48be325dad50098ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:48:38.488083 18989 ts_tablet_manager.cc:1939] T bd521a8d4573405a9e815b43057197aa P a41ccac654b7422eb5427e6014b48898: Deleting consensus metadata
I20260926 14:48:38.488346 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 18686
I20260926 14:48:38.493566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 18820
I20260926 14:48:38.494681 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.493631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40253 (local address 127.0.0.1:41396)
0926 14:48:38.494512 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:38.494591 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40253: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":801}
W20260926 14:48:38.495090 18476 catalog_manager.cc:4584] TS b7248a99679b479484b3cb9edb249899 (127.0.98.1:40253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd521a8d4573405a9e815b43057197aa: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40253: connect: Connection refused (error 111)
I20260926 14:48:38.495446 18961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.495236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36879 (local address 127.0.0.1:37048)
0926 14:48:38.495339 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:38.495389 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:48:38.498989 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 18458
2026-09-26T14:48:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2348 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T14:48:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:38Z Disabled control of system clock
I20260926 14:48:38.528443 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33445
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33445
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:48:38.639066 18996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:38.639357 18996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:38.639392 18996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:38.643564 18996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:38.643628 18996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:38.643642 18996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:38.643659 18996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:38.643673 18996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:38.648682 18996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33445
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:38.649828 18996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:38.651337 18996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:38.659925 19002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:38.659932 19001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:38.659956 19004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:38.660368 18996 server_base.cc:1034] running on GCE node
I20260926 14:48:38.660993 18996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:38.662184 18996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:38.663347 18996 hybrid_clock.cc:648] HybridClock initialized: now 1790434118663328 us; error 34 us; skew 500 ppm
I20260926 14:48:38.665300 18996 webserver.cc:460] Webserver started at http://127.0.98.62:33557/ using document root <none> and password file <none>
I20260926 14:48:38.665884 18996 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:38.665953 18996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:38.666154 18996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:38.668285 18996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "4460dee4873e4b4eb279c6fa48b94b87"
format_stamp: "Formatted at 2026-09-26 14:48:38 on dist-test-slave-s7k6"
server_key: "2f733d007588c729262dc203776410d4"
server_key_iv: "fc882df6a0f39267f48c621984b63e8f"
server_key_version: "encryptionkey@0"
I20260926 14:48:38.668831 18996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "4460dee4873e4b4eb279c6fa48b94b87"
format_stamp: "Formatted at 2026-09-26 14:48:38 on dist-test-slave-s7k6"
server_key: "2f733d007588c729262dc203776410d4"
server_key_iv: "fc882df6a0f39267f48c621984b63e8f"
server_key_version: "encryptionkey@0"
I20260926 14:48:38.672610 18996 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:48:38.675175 19010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:38.676470 18996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:48:38.676595 18996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "4460dee4873e4b4eb279c6fa48b94b87"
format_stamp: "Formatted at 2026-09-26 14:48:38 on dist-test-slave-s7k6"
server_key: "2f733d007588c729262dc203776410d4"
server_key_iv: "fc882df6a0f39267f48c621984b63e8f"
server_key_version: "encryptionkey@0"
I20260926 14:48:38.676717 18996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:38.687516 18996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:38.688182 18996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:38.688344 18996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:38.694868 18996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33445
I20260926 14:48:38.694875 19062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33445 every 8 connection(s)
I20260926 14:48:38.695817 18996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:38.699020 19063 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:38.703907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 18996
I20260926 14:48:38.704046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:38.704386 392 external_mini_cluster.cc:1428] Setting key 0559172a5fa2ed030c07e8295d4e3afe
I20260926 14:48:38.705169 19063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87: Bootstrap starting.
I20260926 14:48:38.707669 19063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:38.708403 19063 log.cc:824] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:38.710237 19063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87: No bootstrap required, opened a new log
I20260926 14:48:38.713903 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.705879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55190 (local address 127.0.98.62:33445)
0926 14:48:38.706612 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:38.706627 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:38.706671 (+ 44us) server_negotiation.cc:407] Connection header received
0926 14:48:38.707371 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:38.707389 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:38.707837 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:38.708176 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:38.708658 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.709635 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.710570 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.711114 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.711284 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.712125 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:38.712130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:38.712144 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:38.712543 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:38.712604 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:38.712625 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:38.712712 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:38.712848 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 14:48:38.713125 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":135,"threads_started":1}
I20260926 14:48:38.713897 19063 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4460dee4873e4b4eb279c6fa48b94b87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33445 } }
I20260926 14:48:38.714171 19063 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:38.714236 19063 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4460dee4873e4b4eb279c6fa48b94b87, State: Initialized, Role: FOLLOWER
I20260926 14:48:38.714680 19063 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4460dee4873e4b4eb279c6fa48b94b87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33445 } }
I20260926 14:48:38.714849 19063 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:38.714910 19063 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:38.715009 19063 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:38.716113 19063 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4460dee4873e4b4eb279c6fa48b94b87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33445 } }
I20260926 14:48:38.716472 19063 leader_election.cc:302] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4460dee4873e4b4eb279c6fa48b94b87; no voters:
I20260926 14:48:38.716745 19063 leader_election.cc:288] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:38.716836 19068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:38.717006 19068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [term 1 LEADER]: Becoming Leader. State: Replica: 4460dee4873e4b4eb279c6fa48b94b87, State: Running, Role: LEADER
I20260926 14:48:38.717316 19068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4460dee4873e4b4eb279c6fa48b94b87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33445 } }
I20260926 14:48:38.718016 19063 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:38.719053 19069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4460dee4873e4b4eb279c6fa48b94b87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4460dee4873e4b4eb279c6fa48b94b87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33445 } } }
I20260926 14:48:38.719197 19069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:38.719489 19070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4460dee4873e4b4eb279c6fa48b94b87. Latest consensus state: current_term: 1 leader_uuid: "4460dee4873e4b4eb279c6fa48b94b87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4460dee4873e4b4eb279c6fa48b94b87" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33445 } } }
I20260926 14:48:38.719583 19070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:38.719682 19077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:38.722638 19077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:38.728374 19077 catalog_manager.cc:1347] Generated new cluster ID: ff085f882ab744939a70f29ef20f2f9c
I20260926 14:48:38.728447 19077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:38.754798 19077 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:38.755843 19077 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:38.765242 19077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87: Generated new TSK 0
I20260926 14:48:38.765988 19077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:38.770867 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33445
--builtin_ntp_servers=127.0.98.20:36983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:48:38.885281 19087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:38.885546 19087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:38.885586 19087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:38.889685 19087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:38.889750 19087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:38.889806 19087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:48:38.894660 19087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:38.895846 19087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:38.897274 19087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:38.907267 19092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:38.907405 19087 server_base.cc:1034] running on GCE node
W20260926 14:48:38.907343 19095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:38.907305 19093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:38.907990 19087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:38.908655 19087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:38.909808 19087 hybrid_clock.cc:648] HybridClock initialized: now 1790434118909781 us; error 48 us; skew 500 ppm
I20260926 14:48:38.911815 19087 webserver.cc:460] Webserver started at http://127.0.98.1:35061/ using document root <none> and password file <none>
I20260926 14:48:38.912403 19087 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:38.912474 19087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:38.912663 19087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:38.914726 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "072cb9ecc70d49549bdf3a759b4cd24a"
format_stamp: "Formatted at 2026-09-26 14:48:38 on dist-test-slave-s7k6"
server_key: "fc90639dd88e2aab031f036dc9d59942"
server_key_iv: "04c781b2e42ba24ddadefc2234a4a085"
server_key_version: "encryptionkey@0"
I20260926 14:48:38.915333 19087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "072cb9ecc70d49549bdf3a759b4cd24a"
format_stamp: "Formatted at 2026-09-26 14:48:38 on dist-test-slave-s7k6"
server_key: "fc90639dd88e2aab031f036dc9d59942"
server_key_iv: "04c781b2e42ba24ddadefc2234a4a085"
server_key_version: "encryptionkey@0"
I20260926 14:48:38.919250 19087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:48:38.921832 19101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:38.923237 19087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:38.923357 19087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "072cb9ecc70d49549bdf3a759b4cd24a"
format_stamp: "Formatted at 2026-09-26 14:48:38 on dist-test-slave-s7k6"
server_key: "fc90639dd88e2aab031f036dc9d59942"
server_key_iv: "04c781b2e42ba24ddadefc2234a4a085"
server_key_version: "encryptionkey@0"
I20260926 14:48:38.923462 19087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:38.935248 19087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:38.935890 19087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:38.936053 19087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:38.936616 19087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:38.937561 19087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:38.937616 19087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:38.937668 19087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:38.937697 19087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:38.946453 19087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32909
I20260926 14:48:38.946543 19214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32909 every 8 connection(s)
I20260926 14:48:38.947453 19087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:48:38.956568 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.949393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36421 (local address 127.0.98.62:33445)
0926 14:48:38.949543 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:38.949547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:38.950665 (+ 1118us) server_negotiation.cc:407] Connection header received
0926 14:48:38.952095 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:38.952099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:38.952150 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:38.952230 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:38.953129 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.953678 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.954820 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.955087 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.955172 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.956172 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:38.956175 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:38.956177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:38.956190 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:38.956212 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:38.956219 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:38.956276 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:38.956387 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 14:48:38.956443 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 14:48:38.957358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 19087
I20260926 14:48:38.957470 19217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:38.949814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33445 (local address 127.0.98.1:36421)
0926 14:48:38.950508 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:38.950551 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:38.951861 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:38.952389 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:38.952396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:38.952487 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:38.952992 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.953005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.953769 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.953772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.954670 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:38.954680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:38.955212 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:38.955214 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:38.955360 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:38.955905 (+ 545us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:48:38.955907 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:38.955948 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:48:38.956014 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:38.956281 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:38.956286 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:38.956290 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:48:38.956378 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 14:48:38.956603 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":112,"threads_started":1}
I20260926 14:48:38.957527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:48:38.957823 392 external_mini_cluster.cc:1428] Setting key d6ba49b7f2a4008129352947e3ffb368
I20260926 14:48:38.958812 19215 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33445
I20260926 14:48:38.959069 19215 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:38.959651 19215 heartbeater.cc:511] Master 127.0.98.62:33445 requested a full tablet report, sending...
I20260926 14:48:38.959719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33445
--builtin_ntp_servers=127.0.98.20:36983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:48:38.961264 19027 ts_manager.cc:194] Registered new tserver with Master: 072cb9ecc70d49549bdf3a759b4cd24a (127.0.98.1:32909)
I20260926 14:48:38.962376 19027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:36421
W20260926 14:48:39.074208 19218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:39.074493 19218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:39.074549 19218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:39.079093 19218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:39.079156 19218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:39.079236 19218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:48:39.084362 19218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:39.085531 19218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:39.086978 19218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:39.097043 19226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:39.097093 19223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:39.097110 19224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:39.097153 19218 server_base.cc:1034] running on GCE node
I20260926 14:48:39.097822 19218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:39.098572 19218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:39.099772 19218 hybrid_clock.cc:648] HybridClock initialized: now 1790434119099717 us; error 77 us; skew 500 ppm
I20260926 14:48:39.101856 19218 webserver.cc:460] Webserver started at http://127.0.98.2:34349/ using document root <none> and password file <none>
I20260926 14:48:39.102468 19218 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:39.102543 19218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:39.102780 19218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:39.104993 19218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "c4bd95c719a34c5599128fc9c4360f6d"
format_stamp: "Formatted at 2026-09-26 14:48:39 on dist-test-slave-s7k6"
server_key: "f8cb11c1a0ec91b6a6ece16e778daf84"
server_key_iv: "3e954eaf38b08cf189e5a9f77ad990b8"
server_key_version: "encryptionkey@0"
I20260926 14:48:39.105561 19218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c4bd95c719a34c5599128fc9c4360f6d"
format_stamp: "Formatted at 2026-09-26 14:48:39 on dist-test-slave-s7k6"
server_key: "f8cb11c1a0ec91b6a6ece16e778daf84"
server_key_iv: "3e954eaf38b08cf189e5a9f77ad990b8"
server_key_version: "encryptionkey@0"
I20260926 14:48:39.109501 19218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 14:48:39.112159 19232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:39.113448 19218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:48:39.113569 19218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "c4bd95c719a34c5599128fc9c4360f6d"
format_stamp: "Formatted at 2026-09-26 14:48:39 on dist-test-slave-s7k6"
server_key: "f8cb11c1a0ec91b6a6ece16e778daf84"
server_key_iv: "3e954eaf38b08cf189e5a9f77ad990b8"
server_key_version: "encryptionkey@0"
I20260926 14:48:39.113677 19218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:39.124714 19218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:39.125411 19218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:39.125583 19218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:39.126178 19218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:39.127218 19218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:39.127269 19218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:39.127321 19218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:39.127347 19218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:39.136557 19218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35729
I20260926 14:48:39.136622 19345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35729 every 8 connection(s)
I20260926 14:48:39.137497 19218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:48:39.146091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 19218
I20260926 14:48:39.146189 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:48:39.146553 392 external_mini_cluster.cc:1428] Setting key d2e13beb8ac6bb9c8cc6cb445da785ae
I20260926 14:48:39.147490 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.139355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34981 (local address 127.0.98.62:33445)
0926 14:48:39.139579 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:39.139583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:39.140729 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 14:48:39.142189 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:39.142192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:39.142247 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:39.142354 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:39.143470 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.144027 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.145360 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.145599 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.145691 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.146977 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:39.146980 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:39.146982 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:39.146995 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:39.147017 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:39.147025 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:39.147099 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:39.147266 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 14:48:39.147339 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260926 14:48:39.148471 19348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.139856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33445 (local address 127.0.98.2:34981)
0926 14:48:39.140567 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:39.140610 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:39.141928 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:39.142574 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:39.142583 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:39.142682 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:39.143276 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.143291 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.144235 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.144238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.145226 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.145236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.145771 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.145775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.145945 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.146659 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:48:39.146662 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:39.146714 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:48:39.146820 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:39.147116 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:39.147124 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:39.147130 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:48:39.147255 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 14:48:39.147569 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":114,"threads_started":1}
I20260926 14:48:39.148526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33445
--builtin_ntp_servers=127.0.98.20:36983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:48:39.149668 19346 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33445
I20260926 14:48:39.149915 19346 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:39.150445 19346 heartbeater.cc:511] Master 127.0.98.62:33445 requested a full tablet report, sending...
I20260926 14:48:39.151499 19027 ts_manager.cc:194] Registered new tserver with Master: c4bd95c719a34c5599128fc9c4360f6d (127.0.98.2:35729)
I20260926 14:48:39.152072 19027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:34981
W20260926 14:48:39.262126 19349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:39.262432 19349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:39.262488 19349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:39.266896 19349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:39.266960 19349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:39.267037 19349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:48:39.272080 19349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33445
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:39.273202 19349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:39.274688 19349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:39.284564 19354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:39.284578 19355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:39.284585 19357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:39.284859 19349 server_base.cc:1034] running on GCE node
I20260926 14:48:39.285295 19349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:39.286000 19349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:39.287165 19349 hybrid_clock.cc:648] HybridClock initialized: now 1790434119287115 us; error 65 us; skew 500 ppm
I20260926 14:48:39.289078 19349 webserver.cc:460] Webserver started at http://127.0.98.3:37903/ using document root <none> and password file <none>
I20260926 14:48:39.289685 19349 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:39.289757 19349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:39.290007 19349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:39.292303 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "8085d2c4070b452bb8704b9c0592e9eb"
format_stamp: "Formatted at 2026-09-26 14:48:39 on dist-test-slave-s7k6"
server_key: "bd7097dbef4adef72e416bcc37824d52"
server_key_iv: "fb76dc0e15ff6d9d00535ced883ae0c9"
server_key_version: "encryptionkey@0"
I20260926 14:48:39.292870 19349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8085d2c4070b452bb8704b9c0592e9eb"
format_stamp: "Formatted at 2026-09-26 14:48:39 on dist-test-slave-s7k6"
server_key: "bd7097dbef4adef72e416bcc37824d52"
server_key_iv: "fb76dc0e15ff6d9d00535ced883ae0c9"
server_key_version: "encryptionkey@0"
I20260926 14:48:39.296712 19349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:48:39.299423 19363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:39.300859 19349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:39.300990 19349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "8085d2c4070b452bb8704b9c0592e9eb"
format_stamp: "Formatted at 2026-09-26 14:48:39 on dist-test-slave-s7k6"
server_key: "bd7097dbef4adef72e416bcc37824d52"
server_key_iv: "fb76dc0e15ff6d9d00535ced883ae0c9"
server_key_version: "encryptionkey@0"
I20260926 14:48:39.301118 19349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:39.325608 19349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:39.326355 19349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:39.326545 19349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:39.327282 19349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:39.328313 19349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:39.328368 19349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:39.328428 19349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:39.328455 19349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:39.337277 19349 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34141
I20260926 14:48:39.337317 19476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34141 every 8 connection(s)
I20260926 14:48:39.338222 19349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:48:39.344907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 19349
I20260926 14:48:39.345049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:48:39.345355 392 external_mini_cluster.cc:1428] Setting key 975abdf1c560f4dd046b41e61da86778
I20260926 14:48:39.348464 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.340243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43545 (local address 127.0.98.62:33445)
0926 14:48:39.340379 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:39.340384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:39.341553 (+ 1169us) server_negotiation.cc:407] Connection header received
0926 14:48:39.343085 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:39.343088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:39.343141 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:39.343220 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:39.344198 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.344759 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.346268 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.346517 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.346610 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.347950 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:39.347953 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:39.347955 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:39.347968 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:39.347990 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:39.347999 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:39.348061 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:39.348227 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 14:48:39.348294 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:48:39.349391 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.340654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33445 (local address 127.0.98.3:43545)
0926 14:48:39.341398 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:39.341445 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:39.342857 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:39.343398 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:39.343405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:39.343508 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:39.344068 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.344082 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.344905 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.344909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.346140 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.346153 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.346669 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.346673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.346923 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.347595 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:48:39.347597 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:39.347644 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:48:39.347718 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:39.348076 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:39.348087 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:39.348093 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 14:48:39.348215 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 14:48:39.348546 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":129,"threads_started":1}
I20260926 14:48:39.350484 19477 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33445
I20260926 14:48:39.350718 19477 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:39.351295 19477 heartbeater.cc:511] Master 127.0.98.62:33445 requested a full tablet report, sending...
I20260926 14:48:39.352725 19027 ts_manager.cc:194] Registered new tserver with Master: 8085d2c4070b452bb8704b9c0592e9eb (127.0.98.3:34141)
I20260926 14:48:39.353324 19027 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:43545
I20260926 14:48:39.359495 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:48:39.365406 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.361478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.0.98.62:33445)
0926 14:48:39.361620 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:39.361625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:39.361747 (+ 122us) server_negotiation.cc:407] Connection header received
0926 14:48:39.361857 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:39.361859 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:39.361907 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:39.361976 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:39.362196 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.362708 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.363553 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.363799 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.363930 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.364761 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:39.364764 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:39.364766 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:39.364791 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:39.364813 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:39.364821 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:39.364925 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:39.365126 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 14:48:39.365183 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:48:39.368557 19027 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:48:39.371713 19027 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:48:39.386883 19491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.380733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32909 (local address 127.0.0.1:37060)
0926 14:48:39.381795 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:39.381864 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:39.382007 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:39.382788 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:39.382791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:39.382808 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:39.382890 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.382895 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.384316 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.384319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.385344 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.385355 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.386303 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.386307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.386414 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.386450 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 14:48:39.386560 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 14:48:39.386688 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"mutex_wait_us":19,"thread_start_us":78,"threads_started":1}
I20260926 14:48:39.386883 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.380858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.0.98.1:32909)
0926 14:48:39.381230 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:39.381246 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:39.382150 (+ 904us) server_negotiation.cc:407] Connection header received
0926 14:48:39.382226 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:39.382231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:39.382509 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:39.382622 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:39.383015 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.384180 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.385466 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.386191 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.386290 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.386514 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:39.386600 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 14:48:39.386700 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":125,"threads_started":1}
I20260926 14:48:39.387192 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.380848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34141 (local address 127.0.0.1:40052)
0926 14:48:39.381949 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:39.381962 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:39.382045 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:39.382618 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:39.382625 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:39.382650 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:39.382841 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.382849 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.384291 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.384295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.385634 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.385644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.386739 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.386743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.386894 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.386918 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:48:39.386978 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 14:48:39.387049 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":55,"thread_start_us":113,"threads_started":1}
I20260926 14:48:39.387631 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.381330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35729 (local address 127.0.0.1:55388)
0926 14:48:39.382219 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:39.382232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:39.382303 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:39.383053 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:39.383055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:39.383073 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:39.383156 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.383162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.384116 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.384120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.385633 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.385644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.386739 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.386743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.387349 (+ 606us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.387372 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:48:39.387436 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:48:39.387508 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":544,"thread_start_us":128,"threads_started":1}
I20260926 14:48:39.387734 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.381616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.0.98.2:35729)
0926 14:48:39.382596 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:39.382613 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:39.382634 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:48:39.382706 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:39.382710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:39.382931 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:39.383071 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:39.383270 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.384013 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.385779 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.386600 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.386907 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.387108 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:39.387473 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 14:48:39.387577 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":120,"threads_started":1}
I20260926 14:48:39.388315 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.381662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40052 (local address 127.0.98.3:34141)
0926 14:48:39.382111 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:39.382124 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:39.382144 (+ 20us) server_negotiation.cc:407] Connection header received
0926 14:48:39.382225 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:39.382231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:39.382453 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:39.382616 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:39.382960 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.384170 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.385779 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.386607 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.387790 (+ 1183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.387983 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:39.388027 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 14:48:39.388099 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":152,"threads_started":1}
I20260926 14:48:39.390340 19280 tablet_service.cc:1467] Processing CreateTablet for tablet f0da8674bb2847939be51211c98aee73 (DEFAULT_TABLE table=test-table [id=60f6c2d7f6f84a3aae89e72460b27bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:39.390353 19149 tablet_service.cc:1467] Processing CreateTablet for tablet f0da8674bb2847939be51211c98aee73 (DEFAULT_TABLE table=test-table [id=60f6c2d7f6f84a3aae89e72460b27bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:39.390879 19411 tablet_service.cc:1467] Processing CreateTablet for tablet f0da8674bb2847939be51211c98aee73 (DEFAULT_TABLE table=test-table [id=60f6c2d7f6f84a3aae89e72460b27bf2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:39.391382 19280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0da8674bb2847939be51211c98aee73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:39.391424 19149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0da8674bb2847939be51211c98aee73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:39.391811 19411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0da8674bb2847939be51211c98aee73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:39.397665 19494 tablet_bootstrap.cc:492] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d: Bootstrap starting.
I20260926 14:48:39.399417 19496 tablet_bootstrap.cc:492] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb: Bootstrap starting.
I20260926 14:48:39.400067 19494 tablet_bootstrap.cc:654] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:39.400812 19494 log.cc:824] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:39.401363 19496 tablet_bootstrap.cc:654] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:39.401849 19495 tablet_bootstrap.cc:492] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a: Bootstrap starting.
I20260926 14:48:39.402169 19496 log.cc:824] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:39.402951 19494 tablet_bootstrap.cc:492] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d: No bootstrap required, opened a new log
I20260926 14:48:39.403160 19494 ts_tablet_manager.cc:1397] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:48:39.403851 19495 tablet_bootstrap.cc:654] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:39.404299 19496 tablet_bootstrap.cc:492] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb: No bootstrap required, opened a new log
I20260926 14:48:39.404484 19496 ts_tablet_manager.cc:1397] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:48:39.404740 19495 log.cc:824] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:39.407080 19495 tablet_bootstrap.cc:492] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a: No bootstrap required, opened a new log
I20260926 14:48:39.407272 19495 ts_tablet_manager.cc:1397] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:48:39.407671 19494 raft_consensus.cc:348] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32909 } } peers { permanent_uuid: "8085d2c4070b452bb8704b9c0592e9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "c4bd95c719a34c5599128fc9c4360f6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35729 } }
I20260926 14:48:39.407999 19494 raft_consensus.cc:374] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:39.408061 19494 raft_consensus.cc:729] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4bd95c719a34c5599128fc9c4360f6d, State: Initialized, Role: FOLLOWER
I20260926 14:48:39.408536 19494 consensus_queue.cc:260] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32909 } } peers { permanent_uuid: "8085d2c4070b452bb8704b9c0592e9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "c4bd95c719a34c5599128fc9c4360f6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35729 } }
I20260926 14:48:39.408807 19496 raft_consensus.cc:348] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32909 } } peers { permanent_uuid: "8085d2c4070b452bb8704b9c0592e9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "c4bd95c719a34c5599128fc9c4360f6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35729 } }
I20260926 14:48:39.409055 19496 raft_consensus.cc:374] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:39.409107 19496 raft_consensus.cc:729] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8085d2c4070b452bb8704b9c0592e9eb, State: Initialized, Role: FOLLOWER
I20260926 14:48:39.409476 19346 heartbeater.cc:503] Master 127.0.98.62:33445 was elected leader, sending a full tablet report...
I20260926 14:48:39.409623 19494 ts_tablet_manager.cc:1428] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:48:39.409567 19496 consensus_queue.cc:260] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32909 } } peers { permanent_uuid: "8085d2c4070b452bb8704b9c0592e9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "c4bd95c719a34c5599128fc9c4360f6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35729 } }
I20260926 14:48:39.410175 19477 heartbeater.cc:503] Master 127.0.98.62:33445 was elected leader, sending a full tablet report...
I20260926 14:48:39.410522 19496 ts_tablet_manager.cc:1428] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 14:48:39.413614 19495 raft_consensus.cc:348] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32909 } } peers { permanent_uuid: "8085d2c4070b452bb8704b9c0592e9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "c4bd95c719a34c5599128fc9c4360f6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35729 } }
I20260926 14:48:39.413856 19495 raft_consensus.cc:374] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:39.413906 19495 raft_consensus.cc:729] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 072cb9ecc70d49549bdf3a759b4cd24a, State: Initialized, Role: FOLLOWER
I20260926 14:48:39.414331 19495 consensus_queue.cc:260] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32909 } } peers { permanent_uuid: "8085d2c4070b452bb8704b9c0592e9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "c4bd95c719a34c5599128fc9c4360f6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35729 } }
I20260926 14:48:39.414944 19215 heartbeater.cc:503] Master 127.0.98.62:33445 was elected leader, sending a full tablet report...
I20260926 14:48:39.415213 19495 ts_tablet_manager.cc:1428] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20260926 14:48:39.448652 19216 tablet.cc:2369] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:48:39.486862 19501 raft_consensus.cc:482] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:48:39.487057 19501 raft_consensus.cc:504] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32909 } } peers { permanent_uuid: "8085d2c4070b452bb8704b9c0592e9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "c4bd95c719a34c5599128fc9c4360f6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35729 } }
I20260926 14:48:39.488168 19501 leader_election.cc:288] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 072cb9ecc70d49549bdf3a759b4cd24a (127.0.98.1:32909), c4bd95c719a34c5599128fc9c4360f6d (127.0.98.2:35729)
I20260926 14:48:39.491719 19479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.488374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32909 (local address 127.0.98.3:34195)
0926 14:48:39.488548 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:39.488563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:39.488650 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:39.488914 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:39.488917 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:39.488960 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:39.489058 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.489064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.489808 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.489812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.490608 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.490618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.491308 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.491312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.491409 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.491426 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 14:48:39.491492 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 14:48:39.491568 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":3,"spinlock_wait_cycles":29312}
I20260926 14:48:39.491708 19490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.488494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34195 (local address 127.0.98.1:32909)
0926 14:48:39.488628 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:39.488632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:39.488648 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:48:39.488785 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:39.488788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:39.488830 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:39.488891 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:39.489172 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.489706 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.490743 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.491196 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.491292 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.491353 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:39.491518 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 14:48:39.491590 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:48:39.491991 19503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.488413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35729 (local address 127.0.98.3:55751)
0926 14:48:39.488826 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:39.488837 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:39.488965 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:39.489198 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:39.489200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:39.489215 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:39.489356 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.489362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.490071 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.490075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.490881 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:39.490890 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.491580 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.491583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:39.491688 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.491713 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 14:48:39.491777 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:48:39.491846 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"spinlock_wait_cycles":9216,"thread_start_us":109,"threads_started":1}
I20260926 14:48:39.491992 19492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.488544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55751 (local address 127.0.98.2:35729)
0926 14:48:39.488650 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:39.488655 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:39.488922 (+ 267us) server_negotiation.cc:407] Connection header received
0926 14:48:39.489067 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:39.489069 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:39.489115 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:39.489177 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:39.489476 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.489960 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.491018 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.491437 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.491542 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.491601 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:39.491806 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 14:48:39.491888 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:48:39.492364 19169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0da8674bb2847939be51211c98aee73" candidate_uuid: "8085d2c4070b452bb8704b9c0592e9eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" is_pre_election: true
I20260926 14:48:39.492540 19300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0da8674bb2847939be51211c98aee73" candidate_uuid: "8085d2c4070b452bb8704b9c0592e9eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4bd95c719a34c5599128fc9c4360f6d" is_pre_election: true
I20260926 14:48:39.492671 19169 raft_consensus.cc:2454] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8085d2c4070b452bb8704b9c0592e9eb in term 0.
I20260926 14:48:39.492846 19300 raft_consensus.cc:2454] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8085d2c4070b452bb8704b9c0592e9eb in term 0.
I20260926 14:48:39.493153 19367 leader_election.cc:302] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072cb9ecc70d49549bdf3a759b4cd24a, 8085d2c4070b452bb8704b9c0592e9eb; no voters:
I20260926 14:48:39.493438 19501 raft_consensus.cc:2781] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:48:39.493520 19501 raft_consensus.cc:482] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:48:39.493566 19501 raft_consensus.cc:3037] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:39.494539 19501 raft_consensus.cc:504] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32909 } } peers { permanent_uuid: "8085d2c4070b452bb8704b9c0592e9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "c4bd95c719a34c5599128fc9c4360f6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35729 } }
I20260926 14:48:39.494899 19501 leader_election.cc:288] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [CANDIDATE]: Term 1 election: Requested vote from peers 072cb9ecc70d49549bdf3a759b4cd24a (127.0.98.1:32909), c4bd95c719a34c5599128fc9c4360f6d (127.0.98.2:35729)
I20260926 14:48:39.495226 19169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0da8674bb2847939be51211c98aee73" candidate_uuid: "8085d2c4070b452bb8704b9c0592e9eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "072cb9ecc70d49549bdf3a759b4cd24a"
I20260926 14:48:39.495288 19300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0da8674bb2847939be51211c98aee73" candidate_uuid: "8085d2c4070b452bb8704b9c0592e9eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4bd95c719a34c5599128fc9c4360f6d"
I20260926 14:48:39.495350 19169 raft_consensus.cc:3037] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:39.495424 19300 raft_consensus.cc:3037] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:39.496372 19169 raft_consensus.cc:2454] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8085d2c4070b452bb8704b9c0592e9eb in term 1.
I20260926 14:48:39.496477 19300 raft_consensus.cc:2454] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8085d2c4070b452bb8704b9c0592e9eb in term 1.
I20260926 14:48:39.496701 19367 leader_election.cc:302] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 072cb9ecc70d49549bdf3a759b4cd24a, 8085d2c4070b452bb8704b9c0592e9eb; no voters:
I20260926 14:48:39.496869 19501 raft_consensus.cc:2781] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:39.497116 19501 raft_consensus.cc:686] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 1 LEADER]: Becoming Leader. State: Replica: 8085d2c4070b452bb8704b9c0592e9eb, State: Running, Role: LEADER
I20260926 14:48:39.497437 19501 consensus_queue.cc:237] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32909 } } peers { permanent_uuid: "8085d2c4070b452bb8704b9c0592e9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "c4bd95c719a34c5599128fc9c4360f6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35729 } }
I20260926 14:48:39.501125 19026 catalog_manager.cc:5495] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb reported cstate change: term changed from 0 to 1, leader changed from <none> to 8085d2c4070b452bb8704b9c0592e9eb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8085d2c4070b452bb8704b9c0592e9eb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8085d2c4070b452bb8704b9c0592e9eb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c4bd95c719a34c5599128fc9c4360f6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35729 } health_report { overall_health: UNKNOWN } } }
I20260926 14:48:39.517213 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.513952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40054 (local address 127.0.98.3:34141)
0926 14:48:39.514059 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:39.514062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:39.514074 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:48:39.514160 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:39.514162 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:39.514216 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:39.514301 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:48:39.514499 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.515048 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.515881 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:39.516088 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:39.516165 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:39.516395 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:48:39.516815 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:48:39.516945 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:39.517047 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 14:48:39.517079 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:48:39.523456 19300 raft_consensus.cc:1261] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d [term 1 FOLLOWER]: Refusing update from remote peer 8085d2c4070b452bb8704b9c0592e9eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:39.523463 19169 raft_consensus.cc:1261] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a [term 1 FOLLOWER]: Refusing update from remote peer 8085d2c4070b452bb8704b9c0592e9eb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:39.524336 19504 consensus_queue.cc:1035] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "072cb9ecc70d49549bdf3a759b4cd24a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:48:39.524852 19501 consensus_queue.cc:1035] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4bd95c719a34c5599128fc9c4360f6d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:48:39.535195 19509 mvcc.cc:205] Tried to move back new op lower bound from 7333618153560399872 to 7333618153464754176. Current Snapshot: MvccSnapshot[applied={T|T < 7333618153560399872}]
I20260926 14:48:39.535409 19508 mvcc.cc:205] Tried to move back new op lower bound from 7333618153560399872 to 7333618153464754176. Current Snapshot: MvccSnapshot[applied={T|T < 7333618153560399872}]
I20260926 14:48:39.542300 19027 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:39.542462 19027 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55204:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:39.544845 19027 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4460dee4873e4b4eb279c6fa48b94b87: Sending DeleteTablet for 3 replicas of tablet f0da8674bb2847939be51211c98aee73
I20260926 14:48:39.545662 19149 tablet_service.cc:1514] Processing DeleteTablet for tablet f0da8674bb2847939be51211c98aee73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:39 UTC) from {username='slave'} at 127.0.0.1:37060
I20260926 14:48:39.545720 19280 tablet_service.cc:1514] Processing DeleteTablet for tablet f0da8674bb2847939be51211c98aee73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:39 UTC) from {username='slave'} at 127.0.0.1:55388
I20260926 14:48:39.546248 19517 tablet_replica.cc:330] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d: stopping tablet replica
I20260926 14:48:39.546218 19411 tablet_service.cc:1514] Processing DeleteTablet for tablet f0da8674bb2847939be51211c98aee73 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:39 UTC) from {username='slave'} at 127.0.0.1:40052
I20260926 14:48:39.546442 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 19087
I20260926 14:48:39.546583 19518 tablet_replica.cc:330] T f0da8674bb2847939be51211c98aee73 P 072cb9ecc70d49549bdf3a759b4cd24a: stopping tablet replica
I20260926 14:48:39.547201 19517 raft_consensus.cc:2229] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:48:39.547642 19517 raft_consensus.cc:2258] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:39.549152 19517 ts_tablet_manager.cc:1905] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:48:39.551971 19517 ts_tablet_manager.cc:1918] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:48:39.552073 19517 log.cc:1196] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/f0da8674bb2847939be51211c98aee73
I20260926 14:48:39.552354 19517 ts_tablet_manager.cc:1939] T f0da8674bb2847939be51211c98aee73 P c4bd95c719a34c5599128fc9c4360f6d: Deleting consensus metadata
I20260926 14:48:39.552484 19519 tablet_replica.cc:330] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb: stopping tablet replica
I20260926 14:48:39.552851 19519 raft_consensus.cc:2229] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:48:39.553320 19519 raft_consensus.cc:2258] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:39.553593 19011 catalog_manager.cc:4841] TS c4bd95c719a34c5599128fc9c4360f6d (127.0.98.2:35729): tablet f0da8674bb2847939be51211c98aee73 (table test-table [id=60f6c2d7f6f84a3aae89e72460b27bf2]) successfully deleted
W20260926 14:48:39.554176 19014 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32909 (error 108)
I20260926 14:48:39.554795 19519 ts_tablet_manager.cc:1905] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:48:39.555146 19488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:39.554856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32909 (local address 127.0.0.1:37072)
0926 14:48:39.554981 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:39.555061 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 14:48:39.555212 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 19218
W20260926 14:48:39.557109 19014 catalog_manager.cc:4584] TS 072cb9ecc70d49549bdf3a759b4cd24a (127.0.98.1:32909): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f0da8674bb2847939be51211c98aee73: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32909: connect: Connection refused (error 111)
I20260926 14:48:39.557896 19519 ts_tablet_manager.cc:1918] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:48:39.557992 19519 log.cc:1196] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/f0da8674bb2847939be51211c98aee73
I20260926 14:48:39.558310 19519 ts_tablet_manager.cc:1939] T f0da8674bb2847939be51211c98aee73 P 8085d2c4070b452bb8704b9c0592e9eb: Deleting consensus metadata
I20260926 14:48:39.559487 19013 catalog_manager.cc:4841] TS 8085d2c4070b452bb8704b9c0592e9eb (127.0.98.3:34141): tablet f0da8674bb2847939be51211c98aee73 (table test-table [id=60f6c2d7f6f84a3aae89e72460b27bf2]) successfully deleted
I20260926 14:48:39.561125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 19349
I20260926 14:48:39.566130 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 18996
2026-09-26T14:48:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1066 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:39 dist-test-slave-s7k6 krb5kdc[19523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:39 dist-test-slave-s7k6 krb5kdc[19523](info): set up 2 sockets
Sep 26 14:48:39 dist-test-slave-s7k6 krb5kdc[19523](info): commencing operation
krb5kdc: starting...
W20260926 14:48:40.526598 392 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:40 dist-test-slave-s7k6 krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:48:40.649613 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38821
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:40.765314 19536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:40.765771 19536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:40.765872 19536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:40.770546 19536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:40.770644 19536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:40.770673 19536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:40.770694 19536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:40.770715 19536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:40.776645 19536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:40.777968 19536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:40.779222 19536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:40.788762 19541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:40.788781 19542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:40.788769 19544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:40.788959 19536 server_base.cc:1034] running on GCE node
I20260926 14:48:40.789587 19536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:40.790906 19536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:40.792102 19536 hybrid_clock.cc:648] HybridClock initialized: now 1790434120792076 us; error 46 us; skew 500 ppm
Sep 26 14:48:40 dist-test-slave-s7k6 krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:40.795121 19536 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:48:40.796226 19536 webserver.cc:460] Webserver started at http://127.0.98.62:45721/ using document root <none> and password file <none>
I20260926 14:48:40.796846 19536 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:40.796918 19536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:40.797129 19536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:40.799405 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "326f606fb02948328b98b8c723c9468f"
format_stamp: "Formatted at 2026-09-26 14:48:40 on dist-test-slave-s7k6"
server_key: "89491ca260b5e4df30be59c193bcc412"
server_key_iv: "efee124b2f2b0ca919d09fee54aa514c"
server_key_version: "encryptionkey@0"
I20260926 14:48:40.800030 19536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "326f606fb02948328b98b8c723c9468f"
format_stamp: "Formatted at 2026-09-26 14:48:40 on dist-test-slave-s7k6"
server_key: "89491ca260b5e4df30be59c193bcc412"
server_key_iv: "efee124b2f2b0ca919d09fee54aa514c"
server_key_version: "encryptionkey@0"
I20260926 14:48:40.804133 19536 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:48:40.806977 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:40.808343 19536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:48:40.808488 19536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "326f606fb02948328b98b8c723c9468f"
format_stamp: "Formatted at 2026-09-26 14:48:40 on dist-test-slave-s7k6"
server_key: "89491ca260b5e4df30be59c193bcc412"
server_key_iv: "efee124b2f2b0ca919d09fee54aa514c"
server_key_version: "encryptionkey@0"
I20260926 14:48:40.808612 19536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:40.838722 19536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:40.842207 19536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:40.842410 19536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:40.849849 19536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38821
I20260926 14:48:40.849877 19603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38821 every 8 connection(s)
I20260926 14:48:40.850970 19536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:40.853945 19604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:40.856340 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 19536
I20260926 14:48:40.856496 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:40.856829 392 external_mini_cluster.cc:1428] Setting key a36336884a9fcef51a9473ebb996ee38
I20260926 14:48:40.860883 19604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f: Bootstrap starting.
I20260926 14:48:40.863629 19604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:40.864709 19604 log.cc:824] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:48:40 dist-test-slave-s7k6 krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:48:40.867136 19604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f: No bootstrap required, opened a new log
I20260926 14:48:40.871835 19604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f606fb02948328b98b8c723c9468f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38821 } }
I20260926 14:48:40.872160 19604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:40.872231 19604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 326f606fb02948328b98b8c723c9468f, State: Initialized, Role: FOLLOWER
I20260926 14:48:40.872738 19604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f606fb02948328b98b8c723c9468f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38821 } }
I20260926 14:48:40.872897 19604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:40.872967 19604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:40.873068 19604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:40.873382 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:40.858643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49130 (local address 127.0.98.62:38821)
0926 14:48:40.859186 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:40.859204 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:40.859254 (+ 50us) server_negotiation.cc:407] Connection header received
0926 14:48:40.859972 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:40.859993 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:40.860394 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:40.860850 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:40.861709 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:40.862817 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:40.863924 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:40.864681 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:40.864873 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:40.867706 (+ 2833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:40.867713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:40.867736 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:40.867770 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:40.869891 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:40.870403 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:40.870409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:40.870416 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:40.870598 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:40.870919 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:40.870923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:40.870924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:40.871514 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:40.871733 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:40.871994 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 14:48:40.872321 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":155,"threads_started":1}
I20260926 14:48:40.874153 19604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f606fb02948328b98b8c723c9468f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38821 } }
I20260926 14:48:40.874536 19604 leader_election.cc:302] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 326f606fb02948328b98b8c723c9468f; no voters:
I20260926 14:48:40.874899 19604 leader_election.cc:288] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:40.875000 19609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:40.875228 19609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [term 1 LEADER]: Becoming Leader. State: Replica: 326f606fb02948328b98b8c723c9468f, State: Running, Role: LEADER
I20260926 14:48:40.875563 19609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f606fb02948328b98b8c723c9468f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38821 } }
I20260926 14:48:40.876173 19604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:40.877383 19610 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "326f606fb02948328b98b8c723c9468f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f606fb02948328b98b8c723c9468f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38821 } } }
I20260926 14:48:40.877507 19610 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:40.877597 19611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 326f606fb02948328b98b8c723c9468f. Latest consensus state: current_term: 1 leader_uuid: "326f606fb02948328b98b8c723c9468f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "326f606fb02948328b98b8c723c9468f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38821 } } }
I20260926 14:48:40.877739 19611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:40.877954 19619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:40.881173 19619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:40.887811 19619 catalog_manager.cc:1347] Generated new cluster ID: a1d98e0406184af4b2caa68d50649060
I20260926 14:48:40.887903 19619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:40.897212 19619 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:40.898367 19619 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:40.906454 19619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f: Generated new TSK 0
I20260926 14:48:40.907347 19619 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 14:48:40.934029 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38821
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:41.049698 19630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:41.049986 19630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:41.050030 19630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:41.054167 19630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:41.054250 19630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:41.054312 19630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:48:41.059980 19630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:41.061237 19630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:41.062491 19630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:41.073105 19635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:41.073082 19636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:41.073082 19638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:41.073225 19630 server_base.cc:1034] running on GCE node
I20260926 14:48:41.073771 19630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:41.074388 19630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:41.075577 19630 hybrid_clock.cc:648] HybridClock initialized: now 1790434121075556 us; error 37 us; skew 500 ppm
Sep 26 14:48:41 dist-test-slave-s7k6 krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:41.078404 19630 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 14:48:41.079262 19630 webserver.cc:460] Webserver started at http://127.0.98.1:39319/ using document root <none> and password file <none>
I20260926 14:48:41.079847 19630 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:41.079909 19630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:41.080081 19630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:41.082042 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "f2ffb9200022435e94618849c5678f13"
format_stamp: "Formatted at 2026-09-26 14:48:41 on dist-test-slave-s7k6"
server_key: "17946e341c6427748608f8f1e458c64e"
server_key_iv: "71f30cd97926567608007a50480a45e8"
server_key_version: "encryptionkey@0"
I20260926 14:48:41.082535 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f2ffb9200022435e94618849c5678f13"
format_stamp: "Formatted at 2026-09-26 14:48:41 on dist-test-slave-s7k6"
server_key: "17946e341c6427748608f8f1e458c64e"
server_key_iv: "71f30cd97926567608007a50480a45e8"
server_key_version: "encryptionkey@0"
I20260926 14:48:41.086073 19630 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 14:48:41.088474 19645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:41.089697 19630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:48:41.089789 19630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "f2ffb9200022435e94618849c5678f13"
format_stamp: "Formatted at 2026-09-26 14:48:41 on dist-test-slave-s7k6"
server_key: "17946e341c6427748608f8f1e458c64e"
server_key_iv: "71f30cd97926567608007a50480a45e8"
server_key_version: "encryptionkey@0"
I20260926 14:48:41.089886 19630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:41.116240 19630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:41.119810 19630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:41.119964 19630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:41.120607 19630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:41.121435 19630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:41.121486 19630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:41.121526 19630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:41.121546 19630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:41.130095 19630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43157
I20260926 14:48:41.130129 19758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43157 every 8 connection(s)
I20260926 14:48:41.131078 19630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 14:48:41 dist-test-slave-s7k6 krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:48:41.140887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 19630
I20260926 14:48:41.141011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:48:41.141287 392 external_mini_cluster.cc:1428] Setting key 3dbe441e364e0d5eac22d2dbce72ec64
I20260926 14:48:41.144678 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.132905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34405 (local address 127.0.98.62:38821)
0926 14:48:41.133048 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:41.133051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:41.133926 (+ 875us) server_negotiation.cc:407] Connection header received
0926 14:48:41.134877 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:41.134881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:41.134938 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:41.135020 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:41.136286 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.136822 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.137984 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.138225 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.138313 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.140850 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:41.140854 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:41.140856 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:41.140893 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:41.142543 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:41.143134 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:41.143137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:41.143138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:41.143215 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:41.143551 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:41.143554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:41.143555 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:41.143737 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:41.143821 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:41.144356 (+ 535us) server_negotiation.cc:299] Negotiation successful
0926 14:48:41.144512 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:48:41.145649 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.133304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38821 (local address 127.0.98.1:34405)
0926 14:48:41.133779 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:41.133823 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:41.134633 (+ 810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:41.135193 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:41.135200 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:41.135597 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:41.136131 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.136147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.136940 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.136943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.137861 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.137871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.138366 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.138370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.138544 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.139143 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:48:41.139159 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:41.140648 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:41.142667 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:41.142674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:41.142694 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:41.143032 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:41.143297 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:41.143300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:41.143302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:41.143457 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:41.143833 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:41.143840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:41.144118 (+ 278us) client_negotiation.cc:769] Sending connection context
0926 14:48:41.144288 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 14:48:41.144650 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":118,"threads_started":1}
I20260926 14:48:41.146991 19759 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38821
I20260926 14:48:41.147244 19759 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:41.147801 19759 heartbeater.cc:511] Master 127.0.98.62:38821 requested a full tablet report, sending...
I20260926 14:48:41.149317 19561 ts_manager.cc:194] Registered new tserver with Master: f2ffb9200022435e94618849c5678f13 (127.0.98.1:43157)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 14:48:41.150538 19561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 14:48:41.167121 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38821
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:41.279381 19764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:41.279675 19764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:41.279718 19764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:41.283821 19764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:41.283890 19764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:41.283950 19764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:48:41.289431 19764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:41.290550 19764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:41.291656 19764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:41.301573 19772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:41.301587 19769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:41.301575 19770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:41.301918 19764 server_base.cc:1034] running on GCE node
I20260926 14:48:41.302356 19764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:41.303103 19764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:41.304261 19764 hybrid_clock.cc:648] HybridClock initialized: now 1790434121304236 us; error 41 us; skew 500 ppm
Sep 26 14:48:41 dist-test-slave-s7k6 krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:41.307089 19764 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 14:48:41.308032 19764 webserver.cc:460] Webserver started at http://127.0.98.2:43247/ using document root <none> and password file <none>
I20260926 14:48:41.308619 19764 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:41.308687 19764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:41.308905 19764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:41.311071 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "a5a8c1c99b0945ae974bdc4be4355585"
format_stamp: "Formatted at 2026-09-26 14:48:41 on dist-test-slave-s7k6"
server_key: "54fa1e42fac0e6047d3641bd7a2926f1"
server_key_iv: "696bc3018b8ac1d815a4321835b38ebc"
server_key_version: "encryptionkey@0"
I20260926 14:48:41.311623 19764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a5a8c1c99b0945ae974bdc4be4355585"
format_stamp: "Formatted at 2026-09-26 14:48:41 on dist-test-slave-s7k6"
server_key: "54fa1e42fac0e6047d3641bd7a2926f1"
server_key_iv: "696bc3018b8ac1d815a4321835b38ebc"
server_key_version: "encryptionkey@0"
I20260926 14:48:41.315757 19764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:48:41.318461 19779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:41.319880 19764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:41.319999 19764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "a5a8c1c99b0945ae974bdc4be4355585"
format_stamp: "Formatted at 2026-09-26 14:48:41 on dist-test-slave-s7k6"
server_key: "54fa1e42fac0e6047d3641bd7a2926f1"
server_key_iv: "696bc3018b8ac1d815a4321835b38ebc"
server_key_version: "encryptionkey@0"
I20260926 14:48:41.320132 19764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:41.355530 19764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:41.359324 19764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:41.359513 19764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:41.360170 19764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:41.361017 19764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:41.361068 19764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:41.361109 19764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:41.361123 19764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:41.369596 19764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46683
I20260926 14:48:41.369630 19892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46683 every 8 connection(s)
I20260926 14:48:41.370536 19764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:48:41.373850 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 19764
I20260926 14:48:41.373974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:48:41.374269 392 external_mini_cluster.cc:1428] Setting key 7ed03468d0eacc2e571c6b9750030cdb
Sep 26 14:48:41 dist-test-slave-s7k6 krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 14:48:41.385808 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.372453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38057 (local address 127.0.98.62:38821)
0926 14:48:41.372600 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:41.372605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:41.373464 (+ 859us) server_negotiation.cc:407] Connection header received
0926 14:48:41.374401 (+ 937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:41.374405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:41.374463 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:41.374554 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:41.376231 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.377007 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.378650 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.379078 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.379174 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.382064 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:41.382069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:41.382072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:41.382105 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:41.383696 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:41.384230 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:41.384234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:41.384236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:41.384339 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:41.384681 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:41.384684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:41.384686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:41.384894 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:41.384989 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:41.385561 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 14:48:41.385678 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 14:48:41.387040 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.372854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38821 (local address 127.0.98.2:38057)
0926 14:48:41.373301 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:41.373345 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:41.374178 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:41.374738 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:41.374747 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:41.375371 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:41.376081 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.376100 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.377162 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.377166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.378505 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.378517 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.379234 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.379238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.379450 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.380082 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:48:41.380102 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:41.381836 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:41.383829 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:41.383835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:41.383850 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:41.384122 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:41.384433 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:41.384435 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:41.384436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:41.384581 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:41.385006 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:41.385014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:41.385295 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 14:48:41.385512 (+ 217us) client_negotiation.cc:240] Negotiation successful
0926 14:48:41.385835 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":113,"threads_started":1}
I20260926 14:48:41.388391 19893 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38821
I20260926 14:48:41.388670 19893 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:41.389312 19893 heartbeater.cc:511] Master 127.0.98.62:38821 requested a full tablet report, sending...
I20260926 14:48:41.390529 19561 ts_manager.cc:194] Registered new tserver with Master: a5a8c1c99b0945ae974bdc4be4355585 (127.0.98.2:46683)
I20260926 14:48:41.391149 19561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38057
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 14:48:41.403294 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38821
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:41.517307 19898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:41.517585 19898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:41.517629 19898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:41.521832 19898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:41.521903 19898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:41.521963 19898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:48:41.527412 19898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38821
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:41.528465 19898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:41.529441 19898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:41.539619 19903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:41.539608 19904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:41.539638 19906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:41.539844 19898 server_base.cc:1034] running on GCE node
I20260926 14:48:41.540371 19898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:41.541070 19898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:41.542240 19898 hybrid_clock.cc:648] HybridClock initialized: now 1790434121542218 us; error 38 us; skew 500 ppm
Sep 26 14:48:41 dist-test-slave-s7k6 krb5kdc[19523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:41.545176 19898 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 14:48:41.546146 19898 webserver.cc:460] Webserver started at http://127.0.98.3:36973/ using document root <none> and password file <none>
I20260926 14:48:41.546729 19898 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:41.546836 19898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:41.547091 19898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:41.549254 19898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "b390212d03004ac8a6199e887724bfce"
format_stamp: "Formatted at 2026-09-26 14:48:41 on dist-test-slave-s7k6"
server_key: "3b536f917588e6fdeaf8429835a2c9df"
server_key_iv: "028470e482f60e4b616be5b8a50fc10f"
server_key_version: "encryptionkey@0"
I20260926 14:48:41.549817 19898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b390212d03004ac8a6199e887724bfce"
format_stamp: "Formatted at 2026-09-26 14:48:41 on dist-test-slave-s7k6"
server_key: "3b536f917588e6fdeaf8429835a2c9df"
server_key_iv: "028470e482f60e4b616be5b8a50fc10f"
server_key_version: "encryptionkey@0"
I20260926 14:48:41.553704 19898 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:48:41.556321 19913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:41.557703 19898 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:48:41.557825 19898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "b390212d03004ac8a6199e887724bfce"
format_stamp: "Formatted at 2026-09-26 14:48:41 on dist-test-slave-s7k6"
server_key: "3b536f917588e6fdeaf8429835a2c9df"
server_key_iv: "028470e482f60e4b616be5b8a50fc10f"
server_key_version: "encryptionkey@0"
I20260926 14:48:41.557938 19898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:41.596375 19898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:41.599566 19898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:41.599753 19898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:41.600382 19898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:41.601258 19898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:41.601315 19898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:41.601374 19898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:41.601402 19898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:41.610409 19898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40579
I20260926 14:48:41.610431 20026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40579 every 8 connection(s)
I20260926 14:48:41.611431 19898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:48:41.619997 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 19898
I20260926 14:48:41.620149 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:48:41.620512 392 external_mini_cluster.cc:1428] Setting key 117945bb5fa2ccd7c0d268b21f88e3f5
Sep 26 14:48:41 dist-test-slave-s7k6 krb5kdc[19523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:48:41.625644 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.613339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38717 (local address 127.0.98.62:38821)
0926 14:48:41.613489 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:41.613493 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:41.614609 (+ 1116us) server_negotiation.cc:407] Connection header received
0926 14:48:41.615557 (+ 948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:41.615560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:41.615611 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:41.615688 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:41.616925 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.617480 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.618795 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.619035 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.619127 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.622044 (+ 2917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:41.622047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:41.622049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:41.622076 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:41.623645 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:41.624143 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:41.624147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:41.624149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:41.624232 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:41.624567 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:41.624570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:41.624572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:41.624769 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:41.624854 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:41.625356 (+ 502us) server_negotiation.cc:299] Negotiation successful
0926 14:48:41.625482 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 14:48:41.626519 20029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.613797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38821 (local address 127.0.98.3:38717)
0926 14:48:41.614452 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:41.614497 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:41.615342 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:41.615876 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:41.615883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:41.616257 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:41.616756 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.616771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.617630 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.617634 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.618616 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.618628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.619183 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.619187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.619379 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.619986 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:48:41.620009 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:41.621860 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:41.623752 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:41.623759 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:41.623778 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:41.624039 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:41.624332 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:48:41.624335 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:48:41.624338 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:48:41.624481 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:48:41.624864 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:41.624871 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:41.625129 (+ 258us) client_negotiation.cc:769] Sending connection context
0926 14:48:41.625303 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 14:48:41.625662 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":116,"threads_started":1}
I20260926 14:48:41.627715 20027 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38821
I20260926 14:48:41.627978 20027 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:41.628510 20027 heartbeater.cc:511] Master 127.0.98.62:38821 requested a full tablet report, sending...
I20260926 14:48:41.629648 19561 ts_manager.cc:194] Registered new tserver with Master: b390212d03004ac8a6199e887724bfce (127.0.98.3:40579)
I20260926 14:48:41.630266 19561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38717
I20260926 14:48:41.635004 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:48:41.644228 19607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.636853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49134 (local address 127.0.98.62:38821)
0926 14:48:41.636995 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:41.636999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:41.637180 (+ 181us) server_negotiation.cc:407] Connection header received
0926 14:48:41.637299 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:41.637303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:41.637353 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:41.637415 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:41.638145 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.638673 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.639535 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.639760 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.639848 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.641278 (+ 1430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:41.641281 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:41.641284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:41.641309 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:41.642597 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:41.643045 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:41.643050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:41.643051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:41.643127 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:41.643372 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:41.643375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:41.643377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:41.643601 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:41.643747 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:41.643951 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 14:48:41.644071 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:48:41.647204 19561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:48:41.650286 19561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:48:41.665131 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.659921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.0.98.3:40579)
0926 14:48:41.660639 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:41.660649 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:41.660670 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:48:41.660738 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:41.660742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:41.660903 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:41.661040 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:41.661329 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.662473 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.663885 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.664455 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.664554 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.664719 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:41.664864 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 14:48:41.664965 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":95,"threads_started":1}
I20260926 14:48:41.665130 20038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.659388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40579 (local address 127.0.0.1:53164)
0926 14:48:41.660346 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:41.660357 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:41.660418 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:41.661036 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:41.661040 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:41.661057 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:41.661190 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.661197 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.662647 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.662650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.663765 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.663775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.664573 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.664577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.664681 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.664718 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 14:48:41.664826 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 14:48:41.664967 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":908,"mutex_wait_us":26,"spinlock_wait_cycles":10496,"thread_start_us":95,"threads_started":1}
I20260926 14:48:41.665134 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.659156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43157 (local address 127.0.0.1:58874)
0926 14:48:41.660082 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:41.660096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:41.660185 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:41.660817 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:41.660824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:41.660846 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:41.660994 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.661002 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.662210 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.662213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.663641 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.663652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.664716 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.664720 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.664806 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.664829 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 14:48:41.664874 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 14:48:41.664984 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":28,"thread_start_us":114,"threads_started":1}
I20260926 14:48:41.665324 20039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.659038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46683 (local address 127.0.0.1:48968)
0926 14:48:41.659861 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:41.659920 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:41.660059 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:41.661055 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:41.661057 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:41.661082 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:41.661190 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.661197 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.662620 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.662624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.663817 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.663825 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.664967 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.664970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.665056 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.665094 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 14:48:41.665156 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 14:48:41.665217 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"mutex_wait_us":103,"thread_start_us":76,"threads_started":1}
I20260926 14:48:41.665732 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.659198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.0.98.2:46683)
0926 14:48:41.660446 (+ 1248us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:41.660460 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:41.660482 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:48:41.660554 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:41.660558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:41.660762 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:41.660906 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:41.661314 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.662490 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.663917 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.664679 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.664765 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.664913 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:41.665485 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 14:48:41.665585 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1128,"thread_start_us":102,"threads_started":1}
I20260926 14:48:41.665906 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.659794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58874 (local address 127.0.98.1:43157)
0926 14:48:41.660309 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:41.660316 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:41.660337 (+ 21us) server_negotiation.cc:407] Connection header received
0926 14:48:41.660416 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:41.660421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:41.660655 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:41.661138 (+ 483us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:41.661257 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.662108 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.663767 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.664578 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.665442 (+ 864us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.665639 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:41.665686 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:48:41.665772 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":114,"threads_started":1}
I20260926 14:48:41.668497 19693 tablet_service.cc:1467] Processing CreateTablet for tablet 69cc72ac71d746a3bcfa4ebf5714d89b (DEFAULT_TABLE table=test-table [id=7e42a52eded04f7490b43b98cd8aef7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:41.668593 19961 tablet_service.cc:1467] Processing CreateTablet for tablet 69cc72ac71d746a3bcfa4ebf5714d89b (DEFAULT_TABLE table=test-table [id=7e42a52eded04f7490b43b98cd8aef7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:41.668586 19827 tablet_service.cc:1467] Processing CreateTablet for tablet 69cc72ac71d746a3bcfa4ebf5714d89b (DEFAULT_TABLE table=test-table [id=7e42a52eded04f7490b43b98cd8aef7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:41.669494 19693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69cc72ac71d746a3bcfa4ebf5714d89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:41.669588 19827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69cc72ac71d746a3bcfa4ebf5714d89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:41.669595 19961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 69cc72ac71d746a3bcfa4ebf5714d89b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:41.676124 20044 tablet_bootstrap.cc:492] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13: Bootstrap starting.
I20260926 14:48:41.677419 20046 tablet_bootstrap.cc:492] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce: Bootstrap starting.
I20260926 14:48:41.678395 20044 tablet_bootstrap.cc:654] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:41.679229 20046 tablet_bootstrap.cc:654] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:41.679204 20044 log.cc:824] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:41.679819 20045 tablet_bootstrap.cc:492] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585: Bootstrap starting.
I20260926 14:48:41.680364 20046 log.cc:824] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:41.681345 20044 tablet_bootstrap.cc:492] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13: No bootstrap required, opened a new log
I20260926 14:48:41.681569 20044 ts_tablet_manager.cc:1397] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:48:41.681787 20045 tablet_bootstrap.cc:654] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:41.682248 20046 tablet_bootstrap.cc:492] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce: No bootstrap required, opened a new log
I20260926 14:48:41.682423 20046 ts_tablet_manager.cc:1397] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:48:41.682749 20045 log.cc:824] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:41.684343 20045 tablet_bootstrap.cc:492] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585: No bootstrap required, opened a new log
I20260926 14:48:41.684512 20045 ts_tablet_manager.cc:1397] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 14:48:41.686184 20046 raft_consensus.cc:348] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8c1c99b0945ae974bdc4be4355585" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46683 } } peers { permanent_uuid: "f2ffb9200022435e94618849c5678f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43157 } } peers { permanent_uuid: "b390212d03004ac8a6199e887724bfce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40579 } }
I20260926 14:48:41.686184 20044 raft_consensus.cc:348] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8c1c99b0945ae974bdc4be4355585" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46683 } } peers { permanent_uuid: "f2ffb9200022435e94618849c5678f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43157 } } peers { permanent_uuid: "b390212d03004ac8a6199e887724bfce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40579 } }
I20260926 14:48:41.686461 20046 raft_consensus.cc:374] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:41.686461 20044 raft_consensus.cc:374] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:41.686525 20044 raft_consensus.cc:729] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2ffb9200022435e94618849c5678f13, State: Initialized, Role: FOLLOWER
I20260926 14:48:41.686542 20046 raft_consensus.cc:729] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b390212d03004ac8a6199e887724bfce, State: Initialized, Role: FOLLOWER
I20260926 14:48:41.687057 20044 consensus_queue.cc:260] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8c1c99b0945ae974bdc4be4355585" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46683 } } peers { permanent_uuid: "f2ffb9200022435e94618849c5678f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43157 } } peers { permanent_uuid: "b390212d03004ac8a6199e887724bfce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40579 } }
I20260926 14:48:41.687054 20046 consensus_queue.cc:260] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8c1c99b0945ae974bdc4be4355585" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46683 } } peers { permanent_uuid: "f2ffb9200022435e94618849c5678f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43157 } } peers { permanent_uuid: "b390212d03004ac8a6199e887724bfce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40579 } }
I20260926 14:48:41.687995 19759 heartbeater.cc:503] Master 127.0.98.62:38821 was elected leader, sending a full tablet report...
I20260926 14:48:41.688109 20044 ts_tablet_manager.cc:1428] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:48:41.688108 20046 ts_tablet_manager.cc:1428] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 14:48:41.688642 20045 raft_consensus.cc:348] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8c1c99b0945ae974bdc4be4355585" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46683 } } peers { permanent_uuid: "f2ffb9200022435e94618849c5678f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43157 } } peers { permanent_uuid: "b390212d03004ac8a6199e887724bfce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40579 } }
I20260926 14:48:41.688912 20045 raft_consensus.cc:374] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:41.688966 20045 raft_consensus.cc:729] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a8c1c99b0945ae974bdc4be4355585, State: Initialized, Role: FOLLOWER
I20260926 14:48:41.689262 20027 heartbeater.cc:503] Master 127.0.98.62:38821 was elected leader, sending a full tablet report...
I20260926 14:48:41.689473 20045 consensus_queue.cc:260] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8c1c99b0945ae974bdc4be4355585" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46683 } } peers { permanent_uuid: "f2ffb9200022435e94618849c5678f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43157 } } peers { permanent_uuid: "b390212d03004ac8a6199e887724bfce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40579 } }
I20260926 14:48:41.690191 19893 heartbeater.cc:503] Master 127.0.98.62:38821 was elected leader, sending a full tablet report...
I20260926 14:48:41.691349 20045 ts_tablet_manager.cc:1428] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 14:48:41.763619 20051 raft_consensus.cc:482] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:48:41.763832 20051 raft_consensus.cc:504] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8c1c99b0945ae974bdc4be4355585" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46683 } } peers { permanent_uuid: "f2ffb9200022435e94618849c5678f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43157 } } peers { permanent_uuid: "b390212d03004ac8a6199e887724bfce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40579 } }
I20260926 14:48:41.765026 20051 leader_election.cc:288] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5a8c1c99b0945ae974bdc4be4355585 (127.0.98.2:46683), b390212d03004ac8a6199e887724bfce (127.0.98.3:40579)
I20260926 14:48:41.770380 20040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.765454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42987 (local address 127.0.98.2:46683)
0926 14:48:41.765647 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:41.765650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:41.765753 (+ 103us) server_negotiation.cc:407] Connection header received
0926 14:48:41.765923 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:41.765926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:41.765976 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:41.766051 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:41.766389 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.767222 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.769045 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.769707 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.769811 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.769877 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:41.770174 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 14:48:41.770248 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260926 14:48:41.770380 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.765406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48507 (local address 127.0.98.3:40579)
0926 14:48:41.765534 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:41.765539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:41.765810 (+ 271us) server_negotiation.cc:407] Connection header received
0926 14:48:41.765927 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:41.765929 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:41.765976 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:41.766051 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:41.766389 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.767242 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.769083 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.769698 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.769811 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.769877 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:41.770169 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 14:48:41.770248 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:48:41.770443 20054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.765289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46683 (local address 127.0.98.1:42987)
0926 14:48:41.765660 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:41.765674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:41.765814 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:41.766072 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:41.766075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:41.766111 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:41.766272 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.766279 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.767377 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.767381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.768876 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.768891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.769849 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.769854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.770067 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.770086 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 14:48:41.770147 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 14:48:41.770237 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":110,"spinlock_wait_cycles":4608,"thread_start_us":142,"threads_started":1}
I20260926 14:48:41.770443 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.765301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40579 (local address 127.0.98.1:48507)
0926 14:48:41.765723 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:41.765734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:41.765821 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:41.766072 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:41.766075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:41.766110 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:41.766274 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.766279 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.767377 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.767381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.768876 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:41.768891 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.769849 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.769854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:41.770020 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.770050 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 14:48:41.770135 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:48:41.770238 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":148,"spinlock_wait_cycles":34816,"thread_start_us":118,"threads_started":1}
I20260926 14:48:41.771198 19847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69cc72ac71d746a3bcfa4ebf5714d89b" candidate_uuid: "f2ffb9200022435e94618849c5678f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a8c1c99b0945ae974bdc4be4355585" is_pre_election: true
I20260926 14:48:41.771349 19981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69cc72ac71d746a3bcfa4ebf5714d89b" candidate_uuid: "f2ffb9200022435e94618849c5678f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b390212d03004ac8a6199e887724bfce" is_pre_election: true
I20260926 14:48:41.771509 19847 raft_consensus.cc:2454] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ffb9200022435e94618849c5678f13 in term 0.
I20260926 14:48:41.771648 19981 raft_consensus.cc:2454] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2ffb9200022435e94618849c5678f13 in term 0.
I20260926 14:48:41.772037 19649 leader_election.cc:302] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a8c1c99b0945ae974bdc4be4355585, f2ffb9200022435e94618849c5678f13; no voters:
I20260926 14:48:41.772329 20051 raft_consensus.cc:2781] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:48:41.772415 20051 raft_consensus.cc:482] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:48:41.772464 20051 raft_consensus.cc:3037] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:41.773501 20051 raft_consensus.cc:504] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8c1c99b0945ae974bdc4be4355585" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46683 } } peers { permanent_uuid: "f2ffb9200022435e94618849c5678f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43157 } } peers { permanent_uuid: "b390212d03004ac8a6199e887724bfce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40579 } }
I20260926 14:48:41.773871 20051 leader_election.cc:288] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [CANDIDATE]: Term 1 election: Requested vote from peers a5a8c1c99b0945ae974bdc4be4355585 (127.0.98.2:46683), b390212d03004ac8a6199e887724bfce (127.0.98.3:40579)
I20260926 14:48:41.774299 19847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69cc72ac71d746a3bcfa4ebf5714d89b" candidate_uuid: "f2ffb9200022435e94618849c5678f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5a8c1c99b0945ae974bdc4be4355585"
I20260926 14:48:41.774299 19981 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69cc72ac71d746a3bcfa4ebf5714d89b" candidate_uuid: "f2ffb9200022435e94618849c5678f13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b390212d03004ac8a6199e887724bfce"
I20260926 14:48:41.774423 19981 raft_consensus.cc:3037] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:41.774423 19847 raft_consensus.cc:3037] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:41.775749 19847 raft_consensus.cc:2454] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ffb9200022435e94618849c5678f13 in term 1.
I20260926 14:48:41.775749 19981 raft_consensus.cc:2454] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2ffb9200022435e94618849c5678f13 in term 1.
I20260926 14:48:41.776072 19649 leader_election.cc:302] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5a8c1c99b0945ae974bdc4be4355585, f2ffb9200022435e94618849c5678f13; no voters:
I20260926 14:48:41.776275 20051 raft_consensus.cc:2781] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:41.776494 20051 raft_consensus.cc:686] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 1 LEADER]: Becoming Leader. State: Replica: f2ffb9200022435e94618849c5678f13, State: Running, Role: LEADER
I20260926 14:48:41.776901 20051 consensus_queue.cc:237] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8c1c99b0945ae974bdc4be4355585" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46683 } } peers { permanent_uuid: "f2ffb9200022435e94618849c5678f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43157 } } peers { permanent_uuid: "b390212d03004ac8a6199e887724bfce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40579 } }
I20260926 14:48:41.780543 19565 catalog_manager.cc:5495] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2ffb9200022435e94618849c5678f13 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f2ffb9200022435e94618849c5678f13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8c1c99b0945ae974bdc4be4355585" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46683 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2ffb9200022435e94618849c5678f13" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43157 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b390212d03004ac8a6199e887724bfce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40579 } health_report { overall_health: UNKNOWN } } }
I20260926 14:48:41.817875 20042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.814092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58880 (local address 127.0.98.1:43157)
0926 14:48:41.814241 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:41.814245 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:41.814260 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:48:41.814460 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:41.814463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:41.814519 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:41.814621 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:48:41.814908 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.815456 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.816329 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:41.816547 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:41.816640 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:41.816949 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:48:41.817409 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:48:41.817522 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:41.817637 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 14:48:41.817704 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:48:41.824581 19981 raft_consensus.cc:1261] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce [term 1 FOLLOWER]: Refusing update from remote peer f2ffb9200022435e94618849c5678f13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:41.824581 19847 raft_consensus.cc:1261] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585 [term 1 FOLLOWER]: Refusing update from remote peer f2ffb9200022435e94618849c5678f13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:41.825505 20051 consensus_queue.cc:1035] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5a8c1c99b0945ae974bdc4be4355585" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46683 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:48:41.825775 20055 consensus_queue.cc:1035] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b390212d03004ac8a6199e887724bfce" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:48:41.844336 19561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:41.844501 19561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49134:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:41.846787 19561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 326f606fb02948328b98b8c723c9468f: Sending DeleteTablet for 3 replicas of tablet 69cc72ac71d746a3bcfa4ebf5714d89b
I20260926 14:48:41.847549 19693 tablet_service.cc:1514] Processing DeleteTablet for tablet 69cc72ac71d746a3bcfa4ebf5714d89b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58874
I20260926 14:48:41.847553 19827 tablet_service.cc:1514] Processing DeleteTablet for tablet 69cc72ac71d746a3bcfa4ebf5714d89b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48968
I20260926 14:48:41.847674 19961 tablet_service.cc:1514] Processing DeleteTablet for tablet 69cc72ac71d746a3bcfa4ebf5714d89b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53164
I20260926 14:48:41.848040 20068 tablet_replica.cc:330] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13: stopping tablet replica
I20260926 14:48:41.848364 20068 raft_consensus.cc:2229] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:48:41.848711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 19630
I20260926 14:48:41.848938 20068 raft_consensus.cc:2258] T 69cc72ac71d746a3bcfa4ebf5714d89b P f2ffb9200022435e94618849c5678f13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:41.850844 20070 tablet_replica.cc:330] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce: stopping tablet replica
I20260926 14:48:41.851153 20070 raft_consensus.cc:2229] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:48:41.851454 20070 raft_consensus.cc:2258] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:41.851804 20069 tablet_replica.cc:330] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585: stopping tablet replica
I20260926 14:48:41.852087 20069 raft_consensus.cc:2229] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:48:41.852370 20069 raft_consensus.cc:2258] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:41.853886 20069 ts_tablet_manager.cc:1905] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:48:41.853888 20070 ts_tablet_manager.cc:1905] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 14:48:41.856392 19553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43157 (error 108)
I20260926 14:48:41.856659 20070 ts_tablet_manager.cc:1918] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:48:41.856742 20070 log.cc:1196] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/69cc72ac71d746a3bcfa4ebf5714d89b
I20260926 14:48:41.856720 20069 ts_tablet_manager.cc:1918] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:48:41.856808 20069 log.cc:1196] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/69cc72ac71d746a3bcfa4ebf5714d89b
I20260926 14:48:41.857033 20070 ts_tablet_manager.cc:1939] T 69cc72ac71d746a3bcfa4ebf5714d89b P b390212d03004ac8a6199e887724bfce: Deleting consensus metadata
I20260926 14:48:41.857112 20069 ts_tablet_manager.cc:1939] T 69cc72ac71d746a3bcfa4ebf5714d89b P a5a8c1c99b0945ae974bdc4be4355585: Deleting consensus metadata
I20260926 14:48:41.857453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 19764
I20260926 14:48:41.858372 19554 catalog_manager.cc:4841] TS b390212d03004ac8a6199e887724bfce (127.0.98.3:40579): tablet 69cc72ac71d746a3bcfa4ebf5714d89b (table test-table [id=7e42a52eded04f7490b43b98cd8aef7f]) successfully deleted
I20260926 14:48:41.860188 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.859904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43157 (local address 127.0.0.1:58892)
0926 14:48:41.860039 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:41.860114 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 14:48:41.860419 19553 catalog_manager.cc:4584] TS f2ffb9200022435e94618849c5678f13 (127.0.98.1:43157): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 69cc72ac71d746a3bcfa4ebf5714d89b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43157: connect: Connection refused (error 111)
I20260926 14:48:41.864634 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 19898
I20260926 14:48:41.866482 20041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:41.864803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46683 (local address 127.0.0.1:48972)
0926 14:48:41.866352 (+ 1549us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:41.866411 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46683: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1465}
I20260926 14:48:41.870062 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 19536
2026-09-26T14:48:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2305 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T14:48:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:41Z Disabled control of system clock
I20260926 14:48:41.899000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46845
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:48:42.009104 20075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:42.009392 20075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:42.009431 20075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:42.013831 20075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:42.013906 20075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:42.013919 20075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:42.013939 20075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:42.013954 20075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:42.018918 20075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46845
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:42.020082 20075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:42.021513 20075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:42.029963 20080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:42.029958 20083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:42.029958 20081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:42.030092 20075 server_base.cc:1034] running on GCE node
I20260926 14:48:42.030948 20075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:42.032068 20075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:42.033242 20075 hybrid_clock.cc:648] HybridClock initialized: now 1790434122033214 us; error 45 us; skew 500 ppm
I20260926 14:48:42.035292 20075 webserver.cc:460] Webserver started at http://127.0.98.62:44419/ using document root <none> and password file <none>
I20260926 14:48:42.035859 20075 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:42.035930 20075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:42.036139 20075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:42.038195 20075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "76c56b8444af48298626262505bce73c"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "07a1e349667301df2693707db4c00774"
server_key_iv: "efa4b8259226ced9050ac0415c1c8c37"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.038736 20075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "76c56b8444af48298626262505bce73c"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "07a1e349667301df2693707db4c00774"
server_key_iv: "efa4b8259226ced9050ac0415c1c8c37"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.042582 20075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 14:48:42.045238 20089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:42.046667 20075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:48:42.046844 20075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "76c56b8444af48298626262505bce73c"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "07a1e349667301df2693707db4c00774"
server_key_iv: "efa4b8259226ced9050ac0415c1c8c37"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.046968 20075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:42.068809 20075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:42.069594 20075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:42.069784 20075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:42.076740 20075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46845
I20260926 14:48:42.076784 20141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46845 every 8 connection(s)
I20260926 14:48:42.077730 20075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:42.081140 20142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:42.084978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 20075
I20260926 14:48:42.085105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:42.085412 392 external_mini_cluster.cc:1428] Setting key 2d8bc9634c592bf50cb95a579eea2d5e
I20260926 14:48:42.087407 20142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c: Bootstrap starting.
I20260926 14:48:42.089694 20142 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:42.090500 20142 log.cc:824] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:42.092504 20142 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c: No bootstrap required, opened a new log
I20260926 14:48:42.095072 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.086996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.0.98.62:46845)
0926 14:48:42.087736 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:42.087756 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:42.087794 (+ 38us) server_negotiation.cc:407] Connection header received
0926 14:48:42.088483 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:42.088501 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:42.088978 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:42.089320 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:42.089771 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.090889 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.091782 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.092282 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.092430 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.093259 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:42.093264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:42.093278 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:42.093672 (+ 394us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:42.093737 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:42.093758 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:42.093841 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:42.093979 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 14:48:42.094237 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":172,"threads_started":1}
I20260926 14:48:42.096204 20142 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c56b8444af48298626262505bce73c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } }
I20260926 14:48:42.096489 20142 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:42.096549 20142 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76c56b8444af48298626262505bce73c, State: Initialized, Role: FOLLOWER
I20260926 14:48:42.097007 20142 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c56b8444af48298626262505bce73c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } }
I20260926 14:48:42.097175 20142 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:42.097250 20142 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:42.097337 20142 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:42.098268 20142 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c56b8444af48298626262505bce73c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } }
I20260926 14:48:42.098619 20142 leader_election.cc:302] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76c56b8444af48298626262505bce73c; no voters:
I20260926 14:48:42.098946 20142 leader_election.cc:288] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:42.099044 20147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:42.099257 20147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [term 1 LEADER]: Becoming Leader. State: Replica: 76c56b8444af48298626262505bce73c, State: Running, Role: LEADER
I20260926 14:48:42.099627 20147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c56b8444af48298626262505bce73c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } }
I20260926 14:48:42.100214 20142 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:42.101300 20149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76c56b8444af48298626262505bce73c. Latest consensus state: current_term: 1 leader_uuid: "76c56b8444af48298626262505bce73c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c56b8444af48298626262505bce73c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } } }
I20260926 14:48:42.101441 20149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:42.101725 20148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76c56b8444af48298626262505bce73c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76c56b8444af48298626262505bce73c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46845 } } }
I20260926 14:48:42.101826 20148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:42.101827 20156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:42.105274 20156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:42.111413 20156 catalog_manager.cc:1347] Generated new cluster ID: 692fcb1365964988ab7be7ad494e1ee3
I20260926 14:48:42.111487 20156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:42.119719 20156 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:42.120929 20156 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:42.126859 20156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c: Generated new TSK 0
I20260926 14:48:42.127665 20156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:42.140738 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--builtin_ntp_servers=127.0.98.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 14:48:42.252852 20166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:42.253136 20166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:42.253175 20166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:42.257463 20166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:42.257534 20166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:42.257598 20166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 14:48:42.263034 20166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:42.264263 20166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:42.265880 20166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:42.276814 20172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:42.276818 20174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:42.277002 20166 server_base.cc:1034] running on GCE node
W20260926 14:48:42.276820 20171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:42.277541 20166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:42.278188 20166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:42.279393 20166 hybrid_clock.cc:648] HybridClock initialized: now 1790434122279354 us; error 52 us; skew 500 ppm
I20260926 14:48:42.281373 20166 webserver.cc:460] Webserver started at http://127.0.98.1:33585/ using document root <none> and password file <none>
I20260926 14:48:42.281950 20166 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:42.282018 20166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:42.282205 20166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:42.284360 20166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/data/instance:
uuid: "303eda7afab84a7a9b84d0a81d785d26"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "6c0b76358d7347e4aaf462331f3bbdd8"
server_key_iv: "2a8706a086d24448d5e50c97fbc304a6"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.284931 20166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/wal/instance:
uuid: "303eda7afab84a7a9b84d0a81d785d26"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "6c0b76358d7347e4aaf462331f3bbdd8"
server_key_iv: "2a8706a086d24448d5e50c97fbc304a6"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.288772 20166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:48:42.291533 20180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:42.292888 20166 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:42.293018 20166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/wal
uuid: "303eda7afab84a7a9b84d0a81d785d26"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "6c0b76358d7347e4aaf462331f3bbdd8"
server_key_iv: "2a8706a086d24448d5e50c97fbc304a6"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.293124 20166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:42.313428 20166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:42.314174 20166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:42.314361 20166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:42.315023 20166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:42.316030 20166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:42.316082 20166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:42.316136 20166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:42.316164 20166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:42.324995 20166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36735
I20260926 14:48:42.325044 20293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36735 every 8 connection(s)
I20260926 14:48:42.325956 20166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/data/info.pb
I20260926 14:48:42.327116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 20166
I20260926 14:48:42.327219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-0/wal/instance
I20260926 14:48:42.327493 392 external_mini_cluster.cc:1428] Setting key 46215c1fa7596dce80de4819351197f2
I20260926 14:48:42.329623 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--builtin_ntp_servers=127.0.98.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:48:42.337088 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.327994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35985 (local address 127.0.98.62:46845)
0926 14:48:42.328153 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:42.328157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:42.329793 (+ 1636us) server_negotiation.cc:407] Connection header received
0926 14:48:42.331585 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:42.331589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:42.331655 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:42.331751 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:42.332746 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.333664 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.334955 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.335303 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.335431 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.336566 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:42.336570 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:42.336573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:42.336590 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:42.336616 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:42.336627 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:42.336707 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:42.336843 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 14:48:42.336920 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:48:42.337950 20296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.328508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46845 (local address 127.0.98.1:35985)
0926 14:48:42.329590 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:42.329656 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:42.331356 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:42.331912 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:42.331920 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:42.332018 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:42.332580 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.332595 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.333820 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.333823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.334833 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.334844 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.335464 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.335467 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.335629 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.336281 (+ 652us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:48:42.336283 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:42.336327 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:48:42.336400 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:42.336705 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:42.336710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:42.336714 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:48:42.336823 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 14:48:42.337071 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":143,"threads_started":1}
I20260926 14:48:42.339365 20294 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46845
I20260926 14:48:42.339635 20294 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:42.340303 20294 heartbeater.cc:511] Master 127.0.98.62:46845 requested a full tablet report, sending...
I20260926 14:48:42.342000 20106 ts_manager.cc:194] Registered new tserver with Master: 303eda7afab84a7a9b84d0a81d785d26 (127.0.98.1:36735)
I20260926 14:48:42.343242 20106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:35985
W20260926 14:48:42.441006 20297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:42.441277 20297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:42.441329 20297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:42.445317 20297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:42.445376 20297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:42.445451 20297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 14:48:42.450505 20297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:42.451659 20297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:42.453114 20297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:42.462703 20305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:42.462718 20303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:42.462844 20302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:42.463274 20297 server_base.cc:1034] running on GCE node
I20260926 14:48:42.463709 20297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:42.464383 20297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:42.465557 20297 hybrid_clock.cc:648] HybridClock initialized: now 1790434122465523 us; error 46 us; skew 500 ppm
I20260926 14:48:42.467640 20297 webserver.cc:460] Webserver started at http://127.0.98.2:44043/ using document root <none> and password file <none>
I20260926 14:48:42.468232 20297 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:42.468307 20297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:42.468524 20297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:42.470635 20297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/data/instance:
uuid: "2a69a98a73ed4f7185d6fad0ec812cda"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "9c54d66220988e1a2bd1c3c2cb46b7a0"
server_key_iv: "561ee6e4a33ba7f9c9ff7ef38c0287b2"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.471226 20297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2a69a98a73ed4f7185d6fad0ec812cda"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "9c54d66220988e1a2bd1c3c2cb46b7a0"
server_key_iv: "561ee6e4a33ba7f9c9ff7ef38c0287b2"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.474958 20297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 14:48:42.477398 20311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:42.478622 20297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:48:42.478737 20297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/wal
uuid: "2a69a98a73ed4f7185d6fad0ec812cda"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "9c54d66220988e1a2bd1c3c2cb46b7a0"
server_key_iv: "561ee6e4a33ba7f9c9ff7ef38c0287b2"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.478873 20297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:42.498710 20297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:42.499495 20297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:42.499680 20297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:42.500443 20297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:42.501488 20297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:42.501536 20297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:42.501591 20297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:42.501617 20297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:42.510453 20297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44695
I20260926 14:48:42.510488 20424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44695 every 8 connection(s)
I20260926 14:48:42.511391 20297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/data/info.pb
I20260926 14:48:42.516325 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 20297
I20260926 14:48:42.516446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/wal/instance
I20260926 14:48:42.516705 392 external_mini_cluster.cc:1428] Setting key b67efc480ab2a43001fbe9e8e16c9d8a
I20260926 14:48:42.518622 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--builtin_ntp_servers=127.0.98.20:34219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 14:48:42.521598 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.513260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56697 (local address 127.0.98.62:46845)
0926 14:48:42.513429 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:42.513433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:42.514600 (+ 1167us) server_negotiation.cc:407] Connection header received
0926 14:48:42.516051 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:42.516055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:42.516114 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:42.516211 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:42.517318 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.517930 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.519373 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.519683 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.519798 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.521133 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:42.521137 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:42.521140 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:42.521155 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:42.521178 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:42.521188 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:42.521259 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:42.521389 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 14:48:42.521449 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:48:42.522526 20427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.513685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46845 (local address 127.0.98.2:56697)
0926 14:48:42.514442 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:42.514487 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:42.515830 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:42.516411 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:42.516418 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:42.516540 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:42.517175 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.517196 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.518058 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.518062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.519241 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.519253 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.519900 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.519905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.520125 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.520816 (+ 691us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:48:42.520818 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:42.520877 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:48:42.520964 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:42.521263 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:42.521269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:42.521273 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:48:42.521376 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 14:48:42.521610 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":119,"threads_started":1}
I20260926 14:48:42.523670 20425 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46845
I20260926 14:48:42.523990 20425 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:42.524634 20425 heartbeater.cc:511] Master 127.0.98.62:46845 requested a full tablet report, sending...
I20260926 14:48:42.525836 20106 ts_manager.cc:194] Registered new tserver with Master: 2a69a98a73ed4f7185d6fad0ec812cda (127.0.98.2:44695)
I20260926 14:48:42.526605 20106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:56697
W20260926 14:48:42.634450 20428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:42.634729 20428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:42.634804 20428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:42.638958 20428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:42.639016 20428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:42.639073 20428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 14:48:42.643962 20428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:42.645081 20428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:42.646457 20428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:42.655882 20436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:42.655884 20433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:42.655970 20428 server_base.cc:1034] running on GCE node
W20260926 14:48:42.655882 20434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:42.656599 20428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:42.657235 20428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:42.658401 20428 hybrid_clock.cc:648] HybridClock initialized: now 1790434122658369 us; error 47 us; skew 500 ppm
I20260926 14:48:42.660348 20428 webserver.cc:460] Webserver started at http://127.0.98.3:45353/ using document root <none> and password file <none>
I20260926 14:48:42.661062 20428 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:42.661137 20428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:42.661351 20428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:42.663681 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/data/instance:
uuid: "4667b9c398ca49bf9e3b011f3d400260"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "9cbd4f1e891759fd16da5add01ff91c0"
server_key_iv: "5d1a733d62288bb706af8310a7f5f32d"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.664256 20428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4667b9c398ca49bf9e3b011f3d400260"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "9cbd4f1e891759fd16da5add01ff91c0"
server_key_iv: "5d1a733d62288bb706af8310a7f5f32d"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.668100 20428 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:48:42.670537 20442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:42.671790 20428 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:48:42.671892 20428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/wal
uuid: "4667b9c398ca49bf9e3b011f3d400260"
format_stamp: "Formatted at 2026-09-26 14:48:42 on dist-test-slave-s7k6"
server_key: "9cbd4f1e891759fd16da5add01ff91c0"
server_key_iv: "5d1a733d62288bb706af8310a7f5f32d"
server_key_version: "encryptionkey@0"
I20260926 14:48:42.672003 20428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:42.686563 20428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:42.687300 20428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:42.687469 20428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:42.688055 20428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 14:48:42.689006 20428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 14:48:42.689052 20428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:42.689092 20428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 14:48:42.689121 20428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:42.697772 20428 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46347
I20260926 14:48:42.697801 20555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46347 every 8 connection(s)
I20260926 14:48:42.698717 20428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/data/info.pb
I20260926 14:48:42.705353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 20428
I20260926 14:48:42.705469 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/wal/instance
I20260926 14:48:42.705736 392 external_mini_cluster.cc:1428] Setting key b6976534a33d73d73cf070f72bd5bbea
I20260926 14:48:42.708477 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.700617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34211 (local address 127.0.98.62:46845)
0926 14:48:42.700844 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:42.700848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:42.702070 (+ 1222us) server_negotiation.cc:407] Connection header received
0926 14:48:42.703549 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:42.703552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:42.703611 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:42.703694 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:42.704616 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.705215 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.706491 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.706829 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.706941 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.708077 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:42.708081 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:42.708084 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:42.708098 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:42.708121 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:42.708131 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:42.708184 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:42.708310 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 14:48:42.708362 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260926 14:48:42.709372 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.701208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46845 (local address 127.0.98.3:34211)
0926 14:48:42.701911 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:42.701953 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:42.703333 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:42.703871 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:42.703877 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:42.703968 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:48:42.704474 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.704486 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.705331 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.705334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.706386 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.706396 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.706973 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.706976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.707174 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.707781 (+ 607us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 14:48:42.707783 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:48:42.707839 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 14:48:42.707925 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:48:42.708200 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:48:42.708205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:48:42.708209 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 14:48:42.708302 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 14:48:42.708537 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":140,"threads_started":1}
I20260926 14:48:42.710476 20556 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46845
I20260926 14:48:42.710698 20556 heartbeater.cc:464] Registering TS with master...
I20260926 14:48:42.711232 20556 heartbeater.cc:511] Master 127.0.98.62:46845 requested a full tablet report, sending...
I20260926 14:48:42.712199 20106 ts_manager.cc:194] Registered new tserver with Master: 4667b9c398ca49bf9e3b011f3d400260 (127.0.98.3:46347)
I20260926 14:48:42.712735 20106 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34211
I20260926 14:48:42.719769 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 14:48:42.725419 20145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.721636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.0.98.62:46845)
0926 14:48:42.721748 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:42.721752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:42.721933 (+ 181us) server_negotiation.cc:407] Connection header received
0926 14:48:42.722028 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:42.722030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:42.722077 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:42.722147 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:42.722400 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.723048 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.723809 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.724031 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.724115 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.724975 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:42.724978 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:42.724980 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:42.724992 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:42.725012 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:42.725020 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:42.725084 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:42.725211 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 14:48:42.725271 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:48:42.728243 20106 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 14:48:42.731223 20106 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 14:48:42.745256 20570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.740329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46347 (local address 127.0.0.1:55442)
0926 14:48:42.740909 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:42.740924 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:42.741015 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:42.741456 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:42.741463 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:42.741485 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:42.741634 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.741641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.742696 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.742700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.743931 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.743940 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.744701 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.744704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.744817 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.744860 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 14:48:42.744967 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 14:48:42.745099 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"mutex_wait_us":17,"thread_start_us":97,"threads_started":1}
I20260926 14:48:42.745282 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.740543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.0.98.3:46347)
0926 14:48:42.741005 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:42.741016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:42.741042 (+ 26us) server_negotiation.cc:407] Connection header received
0926 14:48:42.741121 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:42.741127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:42.741310 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:42.741446 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:42.741757 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.742580 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.744049 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.744595 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.744694 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.744894 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:42.745008 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 14:48:42.745102 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":126,"threads_started":1}
I20260926 14:48:42.745913 20567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.739652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44695 (local address 127.0.0.1:56954)
0926 14:48:42.740870 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:42.740881 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:42.740944 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:42.741686 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:42.741689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:42.741705 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:42.741795 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.741801 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.743280 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.743284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.744522 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.744532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.745536 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.745539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.745629 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.745651 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 14:48:42.745707 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 14:48:42.745779 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":36,"thread_start_us":104,"threads_started":1}
I20260926 14:48:42.745983 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.739748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.0.98.2:44695)
0926 14:48:42.740492 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:42.740505 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:42.741064 (+ 559us) server_negotiation.cc:407] Connection header received
0926 14:48:42.741150 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:42.741154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:42.741356 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:42.741499 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:42.741905 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.743140 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.744657 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.745427 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.745524 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.745714 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:42.745761 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 14:48:42.745844 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":103,"threads_started":1}
I20260926 14:48:42.746997 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.740507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.0.98.1:36735)
0926 14:48:42.741248 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:42.741258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:42.741277 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:48:42.741345 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:42.741349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:42.741504 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:42.741628 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:42.741982 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.743172 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.744736 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.745505 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.745610 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.745788 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:42.746721 (+ 933us) server_negotiation.cc:299] Negotiation successful
0926 14:48:42.746843 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":122,"threads_started":1}
I20260926 14:48:42.747429 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:42.739736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36735 (local address 127.0.0.1:34296)
0926 14:48:42.740743 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:42.740800 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:42.740928 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:42.741768 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:42.741771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:42.741788 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:42.741866 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.741871 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.743280 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.743284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.744608 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:42.744617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:42.745850 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:42.745853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:42.745950 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:42.745974 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:48:42.746038 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 14:48:42.746106 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":180,"thread_start_us":90,"threads_started":1}
I20260926 14:48:42.748805 20359 tablet_service.cc:1467] Processing CreateTablet for tablet 25fd0a21471d4255b4113b10ed08617b (DEFAULT_TABLE table=test-table [id=f945fb934858406194c230dacabad925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:42.748824 20490 tablet_service.cc:1467] Processing CreateTablet for tablet 25fd0a21471d4255b4113b10ed08617b (DEFAULT_TABLE table=test-table [id=f945fb934858406194c230dacabad925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:42.749795 20359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fd0a21471d4255b4113b10ed08617b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:42.749848 20490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fd0a21471d4255b4113b10ed08617b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:42.750157 20228 tablet_service.cc:1467] Processing CreateTablet for tablet 25fd0a21471d4255b4113b10ed08617b (DEFAULT_TABLE table=test-table [id=f945fb934858406194c230dacabad925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 14:48:42.751178 20228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25fd0a21471d4255b4113b10ed08617b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:42.756204 20573 tablet_bootstrap.cc:492] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: Bootstrap starting.
I20260926 14:48:42.757471 20574 tablet_bootstrap.cc:492] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: Bootstrap starting.
I20260926 14:48:42.758479 20573 tablet_bootstrap.cc:654] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:42.758491 20575 tablet_bootstrap.cc:492] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26: Bootstrap starting.
I20260926 14:48:42.759274 20573 log.cc:824] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:42.759660 20574 tablet_bootstrap.cc:654] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:42.760515 20575 tablet_bootstrap.cc:654] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:42.760516 20574 log.cc:824] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:42.761154 20573 tablet_bootstrap.cc:492] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: No bootstrap required, opened a new log
I20260926 14:48:42.761317 20575 log.cc:824] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:42.761399 20573 ts_tablet_manager.cc:1397] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:48:42.762676 20574 tablet_bootstrap.cc:492] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: No bootstrap required, opened a new log
I20260926 14:48:42.762902 20574 ts_tablet_manager.cc:1397] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:48:42.763302 20575 tablet_bootstrap.cc:492] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26: No bootstrap required, opened a new log
I20260926 14:48:42.763486 20575 ts_tablet_manager.cc:1397] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 14:48:42.764833 20573 raft_consensus.cc:348] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "303eda7afab84a7a9b84d0a81d785d26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36735 } } peers { permanent_uuid: "4667b9c398ca49bf9e3b011f3d400260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } }
I20260926 14:48:42.765064 20573 raft_consensus.cc:374] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:42.765142 20573 raft_consensus.cc:729] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a69a98a73ed4f7185d6fad0ec812cda, State: Initialized, Role: FOLLOWER
I20260926 14:48:42.765568 20573 consensus_queue.cc:260] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "303eda7afab84a7a9b84d0a81d785d26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36735 } } peers { permanent_uuid: "4667b9c398ca49bf9e3b011f3d400260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } }
I20260926 14:48:42.766182 20425 heartbeater.cc:503] Master 127.0.98.62:46845 was elected leader, sending a full tablet report...
I20260926 14:48:42.766582 20573 ts_tablet_manager.cc:1428] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 14:48:42.767201 20574 raft_consensus.cc:348] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "303eda7afab84a7a9b84d0a81d785d26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36735 } } peers { permanent_uuid: "4667b9c398ca49bf9e3b011f3d400260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } }
I20260926 14:48:42.767475 20574 raft_consensus.cc:374] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:42.767570 20574 raft_consensus.cc:729] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4667b9c398ca49bf9e3b011f3d400260, State: Initialized, Role: FOLLOWER
I20260926 14:48:42.767457 20575 raft_consensus.cc:348] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "303eda7afab84a7a9b84d0a81d785d26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36735 } } peers { permanent_uuid: "4667b9c398ca49bf9e3b011f3d400260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } }
I20260926 14:48:42.767690 20575 raft_consensus.cc:374] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:42.767738 20575 raft_consensus.cc:729] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 303eda7afab84a7a9b84d0a81d785d26, State: Initialized, Role: FOLLOWER
I20260926 14:48:42.768046 20575 consensus_queue.cc:260] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "303eda7afab84a7a9b84d0a81d785d26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36735 } } peers { permanent_uuid: "4667b9c398ca49bf9e3b011f3d400260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } }
I20260926 14:48:42.768023 20574 consensus_queue.cc:260] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "303eda7afab84a7a9b84d0a81d785d26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36735 } } peers { permanent_uuid: "4667b9c398ca49bf9e3b011f3d400260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } }
I20260926 14:48:42.769011 20574 ts_tablet_manager.cc:1428] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 14:48:42.769011 20575 ts_tablet_manager.cc:1428] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 14:48:42.769477 20294 heartbeater.cc:503] Master 127.0.98.62:46845 was elected leader, sending a full tablet report...
I20260926 14:48:42.769519 20556 heartbeater.cc:503] Master 127.0.98.62:46845 was elected leader, sending a full tablet report...
W20260926 14:48:42.827160 20295 tablet.cc:2369] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:48:42.949724 20557 tablet.cc:2369] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:48:43.012468 20426 tablet.cc:2369] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:48:43.109972 20579 raft_consensus.cc:482] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 14:48:43.110179 20579 raft_consensus.cc:504] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "303eda7afab84a7a9b84d0a81d785d26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36735 } } peers { permanent_uuid: "4667b9c398ca49bf9e3b011f3d400260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } }
I20260926 14:48:43.111347 20579 leader_election.cc:288] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 303eda7afab84a7a9b84d0a81d785d26 (127.0.98.1:36735), 4667b9c398ca49bf9e3b011f3d400260 (127.0.98.3:46347)
I20260926 14:48:43.115265 20571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:43.111734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40601 (local address 127.0.98.1:36735)
0926 14:48:43.111879 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:43.111884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:43.112162 (+ 278us) server_negotiation.cc:407] Connection header received
0926 14:48:43.112263 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:43.112267 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:43.112319 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:43.112396 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:43.112636 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:43.113132 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:43.114279 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:43.114660 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:43.114748 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:43.114834 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:43.115062 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 14:48:43.115132 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:48:43.115265 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:43.111821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34869 (local address 127.0.98.3:46347)
0926 14:48:43.111932 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:43.111936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:43.112049 (+ 113us) server_negotiation.cc:407] Connection header received
0926 14:48:43.112169 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:43.112172 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:43.112222 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:43.112293 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 14:48:43.112580 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:43.113111 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:43.114268 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:43.114666 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:43.114788 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:43.114843 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:43.115064 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 14:48:43.115133 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:48:43.115286 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:43.111625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36735 (local address 127.0.98.2:40601)
0926 14:48:43.112061 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:43.112075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:43.112174 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:43.112413 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:43.112415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:43.112429 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:43.112529 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:43.112534 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:43.113235 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:43.113238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:43.114148 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:43.114158 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:43.114796 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:43.114800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:43.114923 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:43.114960 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 14:48:43.115032 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 14:48:43.115124 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":50,"thread_start_us":94,"threads_started":1}
I20260926 14:48:43.115288 20582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:43.111611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46347 (local address 127.0.98.2:34869)
0926 14:48:43.111966 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:43.111978 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:48:43.112075 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:48:43.112309 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:48:43.112312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:48:43.112327 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 14:48:43.112480 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:43.112487 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:43.113211 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:43.113215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:43.114149 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:48:43.114158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:48:43.114796 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:48:43.114800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:48:43.114946 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:43.114970 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 14:48:43.115035 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 14:48:43.115129 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":128,"thread_start_us":111,"threads_started":1}
I20260926 14:48:43.115960 20510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fd0a21471d4255b4113b10ed08617b" candidate_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667b9c398ca49bf9e3b011f3d400260" is_pre_election: true
I20260926 14:48:43.116046 20248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fd0a21471d4255b4113b10ed08617b" candidate_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303eda7afab84a7a9b84d0a81d785d26" is_pre_election: true
I20260926 14:48:43.116261 20510 raft_consensus.cc:2454] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a69a98a73ed4f7185d6fad0ec812cda in term 0.
I20260926 14:48:43.116308 20248 raft_consensus.cc:2454] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a69a98a73ed4f7185d6fad0ec812cda in term 0.
I20260926 14:48:43.116753 20315 leader_election.cc:302] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a69a98a73ed4f7185d6fad0ec812cda, 4667b9c398ca49bf9e3b011f3d400260; no voters:
I20260926 14:48:43.117027 20579 raft_consensus.cc:2781] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 14:48:43.117103 20579 raft_consensus.cc:482] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 14:48:43.117134 20579 raft_consensus.cc:3037] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:43.118132 20579 raft_consensus.cc:504] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "303eda7afab84a7a9b84d0a81d785d26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36735 } } peers { permanent_uuid: "4667b9c398ca49bf9e3b011f3d400260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } }
I20260926 14:48:43.118462 20579 leader_election.cc:288] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [CANDIDATE]: Term 1 election: Requested vote from peers 303eda7afab84a7a9b84d0a81d785d26 (127.0.98.1:36735), 4667b9c398ca49bf9e3b011f3d400260 (127.0.98.3:46347)
I20260926 14:48:43.118844 20248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fd0a21471d4255b4113b10ed08617b" candidate_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "303eda7afab84a7a9b84d0a81d785d26"
I20260926 14:48:43.118844 20510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25fd0a21471d4255b4113b10ed08617b" candidate_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4667b9c398ca49bf9e3b011f3d400260"
I20260926 14:48:43.118970 20510 raft_consensus.cc:3037] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:43.118970 20248 raft_consensus.cc:3037] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:43.120198 20248 raft_consensus.cc:2454] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a69a98a73ed4f7185d6fad0ec812cda in term 1.
I20260926 14:48:43.120198 20510 raft_consensus.cc:2454] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a69a98a73ed4f7185d6fad0ec812cda in term 1.
I20260926 14:48:43.120507 20312 leader_election.cc:302] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a69a98a73ed4f7185d6fad0ec812cda, 303eda7afab84a7a9b84d0a81d785d26, 4667b9c398ca49bf9e3b011f3d400260; no voters:
I20260926 14:48:43.120676 20579 raft_consensus.cc:2781] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:43.120905 20579 raft_consensus.cc:686] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 1 LEADER]: Becoming Leader. State: Replica: 2a69a98a73ed4f7185d6fad0ec812cda, State: Running, Role: LEADER
I20260926 14:48:43.121215 20579 consensus_queue.cc:237] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } } peers { permanent_uuid: "303eda7afab84a7a9b84d0a81d785d26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36735 } } peers { permanent_uuid: "4667b9c398ca49bf9e3b011f3d400260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } }
I20260926 14:48:43.124306 20104 catalog_manager.cc:5495] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a69a98a73ed4f7185d6fad0ec812cda (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a69a98a73ed4f7185d6fad0ec812cda" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "303eda7afab84a7a9b84d0a81d785d26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36735 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4667b9c398ca49bf9e3b011f3d400260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 } health_report { overall_health: UNKNOWN } } }
I20260926 14:48:43.196297 20569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:43.193069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56964 (local address 127.0.98.2:44695)
0926 14:48:43.193198 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:43.193201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:43.193214 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:48:43.193292 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:43.193294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:43.193347 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:43.193433 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:48:43.193625 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:43.194133 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:43.195006 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:43.195206 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:43.195275 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:43.195550 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:48:43.195970 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:48:43.196079 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:43.196139 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 14:48:43.196188 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:48:43.202287 20510 raft_consensus.cc:1261] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260 [term 1 FOLLOWER]: Refusing update from remote peer 2a69a98a73ed4f7185d6fad0ec812cda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:43.202350 20248 raft_consensus.cc:1261] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26 [term 1 FOLLOWER]: Refusing update from remote peer 2a69a98a73ed4f7185d6fad0ec812cda: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:48:43.203203 20584 consensus_queue.cc:1035] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "303eda7afab84a7a9b84d0a81d785d26" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:48:43.203405 20579 consensus_queue.cc:1035] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [LEADER]: Connected to new peer: Peer: permanent_uuid: "4667b9c398ca49bf9e3b011f3d400260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:48:43.211462 20589 mvcc.cc:205] Tried to move back new op lower bound from 7333618168629338112 to 7333618168307351552. Current Snapshot: MvccSnapshot[applied={T|T < 7333618168629338112}]
I20260926 14:48:43.211923 20588 mvcc.cc:205] Tried to move back new op lower bound from 7333618168629338112 to 7333618168307351552. Current Snapshot: MvccSnapshot[applied={T|T < 7333618168629338112}]
I20260926 14:48:43.220111 20106 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:43.220288 20106 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 14:48:43.222681 20106 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 76c56b8444af48298626262505bce73c: Sending DeleteTablet for 3 replicas of tablet 25fd0a21471d4255b4113b10ed08617b
I20260926 14:48:43.223626 20228 tablet_service.cc:1514] Processing DeleteTablet for tablet 25fd0a21471d4255b4113b10ed08617b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:43 UTC) from {username='slave'} at 127.0.0.1:34296
I20260926 14:48:43.223775 20490 tablet_service.cc:1514] Processing DeleteTablet for tablet 25fd0a21471d4255b4113b10ed08617b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:43 UTC) from {username='slave'} at 127.0.0.1:55442
I20260926 14:48:43.224231 20598 tablet_replica.cc:330] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: stopping tablet replica
I20260926 14:48:43.224211 20359 tablet_service.cc:1514] Processing DeleteTablet for tablet 25fd0a21471d4255b4113b10ed08617b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 14:48:43 UTC) from {username='slave'} at 127.0.0.1:56954
I20260926 14:48:43.224609 20598 raft_consensus.cc:2229] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 14:48:43.224709 20597 tablet_replica.cc:330] T 25fd0a21471d4255b4113b10ed08617b P 303eda7afab84a7a9b84d0a81d785d26: stopping tablet replica
I20260926 14:48:43.224974 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 20166
I20260926 14:48:43.224995 20598 raft_consensus.cc:2258] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:43.226953 20598 ts_tablet_manager.cc:1905] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:48:43.227835 20599 tablet_replica.cc:330] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: stopping tablet replica
I20260926 14:48:43.228188 20599 raft_consensus.cc:2229] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 1 LEADER]: Raft consensus shutting down.
I20260926 14:48:43.228654 20599 raft_consensus.cc:2258] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 14:48:43.229782 20598 ts_tablet_manager.cc:1918] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:48:43.229920 20598 log.cc:1196] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-2/wal/wals/25fd0a21471d4255b4113b10ed08617b
I20260926 14:48:43.230237 20598 ts_tablet_manager.cc:1939] T 25fd0a21471d4255b4113b10ed08617b P 4667b9c398ca49bf9e3b011f3d400260: Deleting consensus metadata
I20260926 14:48:43.230243 20599 ts_tablet_manager.cc:1905] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 14:48:43.231518 20093 catalog_manager.cc:4841] TS 4667b9c398ca49bf9e3b011f3d400260 (127.0.98.3:46347): tablet 25fd0a21471d4255b4113b10ed08617b (table test-table [id=f945fb934858406194c230dacabad925]) successfully deleted
W20260926 14:48:43.232558 20090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36735 (error 108)
I20260926 14:48:43.232893 20599 ts_tablet_manager.cc:1918] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 14:48:43.233081 20599 log.cc:1196] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: Deleting WAL directory at /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790433872429009-392-0/minicluster-data/ts-1/wal/wals/25fd0a21471d4255b4113b10ed08617b
I20260926 14:48:43.233377 20599 ts_tablet_manager.cc:1939] T 25fd0a21471d4255b4113b10ed08617b P 2a69a98a73ed4f7185d6fad0ec812cda: Deleting consensus metadata
I20260926 14:48:43.233510 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:43.233233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36735 (local address 127.0.0.1:34300)
0926 14:48:43.233349 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:43.233421 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 14:48:43.233614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 20297
W20260926 14:48:43.234965 20090 catalog_manager.cc:4584] TS 303eda7afab84a7a9b84d0a81d785d26 (127.0.98.1:36735): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 25fd0a21471d4255b4113b10ed08617b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36735: connect: Connection refused (error 111)
I20260926 14:48:43.238734 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 20428
I20260926 14:48:43.239256 20568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:43.238997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44695 (local address 127.0.0.1:56972)
0926 14:48:43.239141 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:48:43.239181 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260926 14:48:43.243376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 20075
2026-09-26T14:48:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1371 ms)
[----------] 6 tests from EncryptionPolicyTest (10371 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:43 dist-test-slave-s7k6 krb5kdc[20603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:43 dist-test-slave-s7k6 krb5kdc[20603](info): set up 2 sockets
Sep 26 14:48:43 dist-test-slave-s7k6 krb5kdc[20603](info): commencing operation
krb5kdc: starting...
W20260926 14:48:44.198664 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:44 dist-test-slave-s7k6 krb5kdc[20603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:44.311309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:44.420688 20616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:44.421005 20616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:44.421065 20616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:44.425303 20616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:44.425383 20616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:44.425408 20616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:44.425429 20616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:44.425457 20616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:44.431144 20616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:44.432221 20616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:44.433257 20616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:44.441574 20622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:44.441614 20624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:44.441594 20621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:44.441982 20616 server_base.cc:1034] running on GCE node
I20260926 14:48:44.442619 20616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:44.443781 20616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:44.444962 20616 hybrid_clock.cc:648] HybridClock initialized: now 1790434124444942 us; error 33 us; skew 500 ppm
Sep 26 14:48:44 dist-test-slave-s7k6 krb5kdc[20603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:44.447865 20616 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:44.448812 20616 webserver.cc:460] Webserver started at http://127.0.0.1:46241/ using document root <none> and password file <none>
I20260926 14:48:44.449383 20616 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:44.449452 20616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:44.449646 20616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:44.451887 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "7aeddf8e8e154b9f82d36f3177c96156"
format_stamp: "Formatted at 2026-09-26 14:48:44 on dist-test-slave-s7k6"
server_key: "73953865d93837a42668716a0aaf5541"
server_key_iv: "fc5ac784f9e23af51ab39e6e5630dee7"
server_key_version: "encryptionkey@0"
I20260926 14:48:44.452466 20616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "7aeddf8e8e154b9f82d36f3177c96156"
format_stamp: "Formatted at 2026-09-26 14:48:44 on dist-test-slave-s7k6"
server_key: "73953865d93837a42668716a0aaf5541"
server_key_iv: "fc5ac784f9e23af51ab39e6e5630dee7"
server_key_version: "encryptionkey@0"
I20260926 14:48:44.456336 20616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:48:44.459075 20631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:44.460457 20616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:48:44.460575 20616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "7aeddf8e8e154b9f82d36f3177c96156"
format_stamp: "Formatted at 2026-09-26 14:48:44 on dist-test-slave-s7k6"
server_key: "73953865d93837a42668716a0aaf5541"
server_key_iv: "fc5ac784f9e23af51ab39e6e5630dee7"
server_key_version: "encryptionkey@0"
I20260926 14:48:44.460682 20616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:44.478418 20616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:44.481742 20616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:44.481909 20616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:44.489439 20616 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37753
I20260926 14:48:44.489467 20683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37753 every 8 connection(s)
I20260926 14:48:44.490464 20616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:44.493498 20684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:44.497421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 20616
I20260926 14:48:44.497581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:44.497891 392 external_mini_cluster.cc:1428] Setting key 59bf124ff3121d8e0c425b4020857f6b
I20260926 14:48:44.499826 20684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156: Bootstrap starting.
I20260926 14:48:44.502193 20684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:44.503037 20684 log.cc:824] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:44.505352 20684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156: No bootstrap required, opened a new log
Sep 26 14:48:44 dist-test-slave-s7k6 krb5kdc[20603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:48:44.509238 20684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeddf8e8e154b9f82d36f3177c96156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } }
I20260926 14:48:44.509482 20684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:44.509541 20684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aeddf8e8e154b9f82d36f3177c96156, State: Initialized, Role: FOLLOWER
I20260926 14:48:44.510004 20684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeddf8e8e154b9f82d36f3177c96156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } }
I20260926 14:48:44.510141 20684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:44.510217 20684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:44.510317 20684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:44.511288 20684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeddf8e8e154b9f82d36f3177c96156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } }
I20260926 14:48:44.511627 20684 leader_election.cc:302] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aeddf8e8e154b9f82d36f3177c96156; no voters:
I20260926 14:48:44.511919 20684 leader_election.cc:288] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:44.512045 20689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:44.512243 20689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [term 1 LEADER]: Becoming Leader. State: Replica: 7aeddf8e8e154b9f82d36f3177c96156, State: Running, Role: LEADER
I20260926 14:48:44.512598 20689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeddf8e8e154b9f82d36f3177c96156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } }
I20260926 14:48:44.512981 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:44.499485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38637 (local address 127.0.0.1:37753)
0926 14:48:44.499945 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:44.499965 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:44.500013 (+ 48us) server_negotiation.cc:407] Connection header received
0926 14:48:44.500748 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:44.500768 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:44.501130 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:44.501465 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:44.502328 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:44.503514 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:44.504838 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:44.505384 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:44.505567 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:44.508033 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:44.508038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:44.508053 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:44.508083 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:44.509910 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:44.510444 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:44.510450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:44.510454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:44.510564 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:44.510929 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:44.510931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:44.510933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:44.511335 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:44.511537 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:44.511855 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 14:48:44.512092 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":143,"threads_started":1}
I20260926 14:48:44.513265 20684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:44.514235 20691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aeddf8e8e154b9f82d36f3177c96156. Latest consensus state: current_term: 1 leader_uuid: "7aeddf8e8e154b9f82d36f3177c96156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeddf8e8e154b9f82d36f3177c96156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } } }
I20260926 14:48:44.514360 20691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:44.514700 20690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aeddf8e8e154b9f82d36f3177c96156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aeddf8e8e154b9f82d36f3177c96156" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37753 } } }
I20260926 14:48:44.514788 20698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:44.514820 20690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:44.516311 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:44.517940 20698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:44.523319 20698 catalog_manager.cc:1347] Generated new cluster ID: 7f2f0559726c4caca75a97add6853798
I20260926 14:48:44.523376 20698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:44.529424 20687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:44.520200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48105 (local address 127.0.0.1:37753)
0926 14:48:44.520308 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:44.520312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:44.520556 (+ 244us) server_negotiation.cc:407] Connection header received
0926 14:48:44.520668 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:44.520671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:44.520715 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:44.520796 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:44.521618 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:44.522436 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:44.523676 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:44.523997 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:44.524100 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:44.525679 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:44.525683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:44.525686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:44.525711 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:44.527300 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:44.527866 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:44.527870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:44.527872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:44.527976 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:44.528356 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:44.528360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:44.528362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:44.528566 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:44.528693 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:44.529150 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 14:48:44.529272 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:48:44.529444 20698 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:44.530714 20698 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:44.540572 20698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aeddf8e8e154b9f82d36f3177c96156: Generated new TSK 0
I20260926 14:48:44.541299 20698 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:44.547894 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 20616
W20260926 14:48:44.553189 20601 connection.cc:535] client connection to 127.0.0.1:37753 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37753): Connection reset by peer (error 104)
2026-09-26T14:48:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:44 dist-test-slave-s7k6 krb5kdc[20715](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:44 dist-test-slave-s7k6 krb5kdc[20715](info): set up 2 sockets
Sep 26 14:48:44 dist-test-slave-s7k6 krb5kdc[20715](info): commencing operation
krb5kdc: starting...
W20260926 14:48:45.502746 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:45 dist-test-slave-s7k6 krb5kdc[20715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:45.620128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:45.730955 20728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:45.731233 20728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:45.731276 20728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:45.735342 20728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:45.735407 20728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:45.735425 20728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:45.735441 20728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:45.735464 20728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:45.740978 20728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:45.742125 20728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:45.743194 20728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:45.751581 20736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:45.751631 20734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:45.751575 20733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:45.752002 20728 server_base.cc:1034] running on GCE node
I20260926 14:48:45.752624 20728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:45.753786 20728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:45.754947 20728 hybrid_clock.cc:648] HybridClock initialized: now 1790434125754927 us; error 36 us; skew 500 ppm
Sep 26 14:48:45 dist-test-slave-s7k6 krb5kdc[20715](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:45.757758 20728 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:45.758649 20728 webserver.cc:460] Webserver started at http://127.0.0.1:41099/ using document root <none> and password file <none>
I20260926 14:48:45.759235 20728 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:45.759307 20728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:45.759521 20728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:45.761608 20728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "9bb54670d39b462a931cb9d2d263ca0a"
format_stamp: "Formatted at 2026-09-26 14:48:45 on dist-test-slave-s7k6"
server_key: "3a818df037728fc782ee67e60ddbfad9"
server_key_iv: "29b99f524cbefe2af9ab6a444d03cfb7"
server_key_version: "encryptionkey@0"
I20260926 14:48:45.762176 20728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "9bb54670d39b462a931cb9d2d263ca0a"
format_stamp: "Formatted at 2026-09-26 14:48:45 on dist-test-slave-s7k6"
server_key: "3a818df037728fc782ee67e60ddbfad9"
server_key_iv: "29b99f524cbefe2af9ab6a444d03cfb7"
server_key_version: "encryptionkey@0"
I20260926 14:48:45.766121 20728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 14:48:45.768735 20743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:45.770051 20728 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:48:45.770195 20728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "9bb54670d39b462a931cb9d2d263ca0a"
format_stamp: "Formatted at 2026-09-26 14:48:45 on dist-test-slave-s7k6"
server_key: "3a818df037728fc782ee67e60ddbfad9"
server_key_iv: "29b99f524cbefe2af9ab6a444d03cfb7"
server_key_version: "encryptionkey@0"
I20260926 14:48:45.770303 20728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:45.791985 20728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:45.795018 20728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:45.795195 20728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:45.801923 20728 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38023
I20260926 14:48:45.801963 20795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38023 every 8 connection(s)
I20260926 14:48:45.802932 20728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:45.805886 20796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:45.806290 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 20728
I20260926 14:48:45.806425 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:45.806720 392 external_mini_cluster.cc:1428] Setting key 10aba7da1d58a5eda8c44dcc27f1d0f3
I20260926 14:48:45.812707 20796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a: Bootstrap starting.
I20260926 14:48:45.815233 20796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:45.815856 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:45.808154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47947 (local address 127.0.0.1:38023)
0926 14:48:45.808641 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:45.808656 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:45.808700 (+ 44us) server_negotiation.cc:407] Connection header received
0926 14:48:45.809393 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:45.809411 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:45.809774 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:45.810099 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:45.810521 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:45.811619 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:45.812478 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:45.812980 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:45.813134 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:45.814109 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:45.814114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:45.814128 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:45.814533 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:45.814592 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:45.814615 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:45.814732 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:45.814952 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 14:48:45.815094 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":146,"threads_started":1}
I20260926 14:48:45.815979 20796 log.cc:824] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:45.818224 20796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a: No bootstrap required, opened a new log
I20260926 14:48:45.819140 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:45.822350 20796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb54670d39b462a931cb9d2d263ca0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } }
I20260926 14:48:45.822605 20796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:45.822659 20796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb54670d39b462a931cb9d2d263ca0a, State: Initialized, Role: FOLLOWER
I20260926 14:48:45.823227 20796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb54670d39b462a931cb9d2d263ca0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } }
I20260926 14:48:45.823400 20796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:45.823468 20796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:45.823571 20796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:45.824754 20796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb54670d39b462a931cb9d2d263ca0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } }
I20260926 14:48:45.825201 20796 leader_election.cc:302] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bb54670d39b462a931cb9d2d263ca0a; no voters:
I20260926 14:48:45.825516 20799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:45.821214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58803 (local address 127.0.0.1:38023)
0926 14:48:45.821330 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:45.821335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:45.821506 (+ 171us) server_negotiation.cc:407] Connection header received
0926 14:48:45.821626 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:45.821629 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:45.821671 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:45.821734 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:45.821997 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:45.822542 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:45.823719 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:45.824018 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:45.824110 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:45.825030 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:45.825034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:45.825037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:45.825050 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:45.825080 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:45.825094 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:45.825172 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:45.825342 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 14:48:45.825383 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:48:45.825603 20796 leader_election.cc:288] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:45.825736 20806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:45.825956 20806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [term 1 LEADER]: Becoming Leader. State: Replica: 9bb54670d39b462a931cb9d2d263ca0a, State: Running, Role: LEADER
I20260926 14:48:45.826355 20806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb54670d39b462a931cb9d2d263ca0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } }
I20260926 14:48:45.826998 20796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:45.827646 20807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bb54670d39b462a931cb9d2d263ca0a. Latest consensus state: current_term: 1 leader_uuid: "9bb54670d39b462a931cb9d2d263ca0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb54670d39b462a931cb9d2d263ca0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } } }
I20260926 14:48:45.827771 20807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:45.828063 20808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bb54670d39b462a931cb9d2d263ca0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb54670d39b462a931cb9d2d263ca0a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38023 } } }
I20260926 14:48:45.828212 20808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:45.828289 20814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:45.831516 20814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:45.837664 20814 catalog_manager.cc:1347] Generated new cluster ID: 68f3d1fa83a5467faf56613b27b36452
I20260926 14:48:45.837724 20814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:45.868821 20814 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:45.869869 20814 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:45.877899 20814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bb54670d39b462a931cb9d2d263ca0a: Generated new TSK 0
I20260926 14:48:45.878592 20814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:45.889392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 20728
W20260926 14:48:45.894291 20713 connection.cc:535] client connection to 127.0.0.1:38023 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38023): Connection reset by peer (error 104)
2026-09-26T14:48:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:45 dist-test-slave-s7k6 krb5kdc[20827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:45 dist-test-slave-s7k6 krb5kdc[20827](info): set up 2 sockets
Sep 26 14:48:45 dist-test-slave-s7k6 krb5kdc[20827](info): commencing operation
krb5kdc: starting...
W20260926 14:48:46.842715 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:46 dist-test-slave-s7k6 krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:46.961067 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34285
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34285
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:47.072235 20840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:47.072539 20840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:47.072603 20840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:47.076794 20840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:47.076869 20840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:47.076894 20840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:47.076915 20840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:47.076942 20840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:47.082369 20840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34285
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:47.083462 20840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:47.084504 20840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:47.093034 20846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:47.093010 20845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:47.093012 20848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:47.093114 20840 server_base.cc:1034] running on GCE node
I20260926 14:48:47.093883 20840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:47.095103 20840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:47.096271 20840 hybrid_clock.cc:648] HybridClock initialized: now 1790434127096254 us; error 32 us; skew 500 ppm
Sep 26 14:48:47 dist-test-slave-s7k6 krb5kdc[20827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:47.099229 20840 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:47.100185 20840 webserver.cc:460] Webserver started at http://127.0.0.1:42495/ using document root <none> and password file <none>
I20260926 14:48:47.100767 20840 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:47.100829 20840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:47.101037 20840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:47.103185 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "7578d97e90dd41fe89476e9ef7884a66"
format_stamp: "Formatted at 2026-09-26 14:48:47 on dist-test-slave-s7k6"
server_key: "d7d2cb87c5be8bc762cfe6500e77b2d7"
server_key_iv: "d869286440d20784ee579b31a6a6ae07"
server_key_version: "encryptionkey@0"
I20260926 14:48:47.103730 20840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "7578d97e90dd41fe89476e9ef7884a66"
format_stamp: "Formatted at 2026-09-26 14:48:47 on dist-test-slave-s7k6"
server_key: "d7d2cb87c5be8bc762cfe6500e77b2d7"
server_key_iv: "d869286440d20784ee579b31a6a6ae07"
server_key_version: "encryptionkey@0"
I20260926 14:48:47.107574 20840 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:48:47.110162 20855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:47.111524 20840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:48:47.111635 20840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "7578d97e90dd41fe89476e9ef7884a66"
format_stamp: "Formatted at 2026-09-26 14:48:47 on dist-test-slave-s7k6"
server_key: "d7d2cb87c5be8bc762cfe6500e77b2d7"
server_key_iv: "d869286440d20784ee579b31a6a6ae07"
server_key_version: "encryptionkey@0"
I20260926 14:48:47.111742 20840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:47.116271 20840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:47.119342 20840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:47.119506 20840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:47.126202 20840 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34285
I20260926 14:48:47.126286 20907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34285 every 8 connection(s)
I20260926 14:48:47.127298 20840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:47.130199 20908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:47.135727 20908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66: Bootstrap starting.
I20260926 14:48:47.136644 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 20840
I20260926 14:48:47.136770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:47.137113 392 external_mini_cluster.cc:1428] Setting key fdf8e1adef94a1ed48e5cc7a245d98fd
I20260926 14:48:47.138581 20908 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:47.139530 20908 log.cc:824] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:47.141786 20908 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66: No bootstrap required, opened a new log
I20260926 14:48:47.142839 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:47.138223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36081 (local address 127.0.0.1:34285)
0926 14:48:47.138729 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:47.138748 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:47.138822 (+ 74us) server_negotiation.cc:407] Connection header received
0926 14:48:47.139709 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:47.139735 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:47.140092 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:47.140479 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:47.141238 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:47.141243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:47.141258 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:47.141269 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:47.141349 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:47.141377 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:47.141494 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:47.141678 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 14:48:47.141849 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":133,"threads_started":1}
I20260926 14:48:47.145740 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:47.146170 20908 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7578d97e90dd41fe89476e9ef7884a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } }
I20260926 14:48:47.146454 20908 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:47.146509 20908 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7578d97e90dd41fe89476e9ef7884a66, State: Initialized, Role: FOLLOWER
I20260926 14:48:47.147027 20908 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7578d97e90dd41fe89476e9ef7884a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } }
I20260926 14:48:47.147204 20908 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:47.147279 20908 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:47.147399 20908 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:47.148530 20908 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7578d97e90dd41fe89476e9ef7884a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } }
I20260926 14:48:47.148917 20908 leader_election.cc:302] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7578d97e90dd41fe89476e9ef7884a66; no voters:
I20260926 14:48:47.149271 20908 leader_election.cc:288] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:47.149330 20918 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:47.149549 20918 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [term 1 LEADER]: Becoming Leader. State: Replica: 7578d97e90dd41fe89476e9ef7884a66, State: Running, Role: LEADER
I20260926 14:48:47.149906 20918 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7578d97e90dd41fe89476e9ef7884a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } }
I20260926 14:48:47.150607 20908 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:47.149556 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:47.148007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56477 (local address 127.0.0.1:34285)
0926 14:48:47.148126 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:47.148130 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:47.148311 (+ 181us) server_negotiation.cc:407] Connection header received
0926 14:48:47.148448 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:47.148450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:47.148481 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:47.148541 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:47.149185 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:47.149188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:47.149189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:47.149199 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:47.149224 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:47.149234 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:47.149310 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:47.149398 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 14:48:47.149440 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:48:47.151139 20918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7578d97e90dd41fe89476e9ef7884a66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7578d97e90dd41fe89476e9ef7884a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } } }
I20260926 14:48:47.151320 20918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:47.151562 20918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7578d97e90dd41fe89476e9ef7884a66. Latest consensus state: current_term: 1 leader_uuid: "7578d97e90dd41fe89476e9ef7884a66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7578d97e90dd41fe89476e9ef7884a66" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34285 } } }
I20260926 14:48:47.151707 20918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:47.152009 20925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:47.154726 20925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:47.160292 20925 catalog_manager.cc:1347] Generated new cluster ID: f562143a56694deda586bda8e0d8aad3
I20260926 14:48:47.160360 20925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:47.175496 20925 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:47.175799 20925 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:47.183122 20925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7578d97e90dd41fe89476e9ef7884a66: Generated new TSK 0
I20260926 14:48:47.184063 20925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:47.192808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 20840
2026-09-26T14:48:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:47 dist-test-slave-s7k6 krb5kdc[20939](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:47 dist-test-slave-s7k6 krb5kdc[20939](info): set up 2 sockets
Sep 26 14:48:47 dist-test-slave-s7k6 krb5kdc[20939](info): commencing operation
krb5kdc: starting...
W20260926 14:48:48.142684 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:48 dist-test-slave-s7k6 krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:48.257447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46497
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46497
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:48.371104 20952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:48.371387 20952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:48.371443 20952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:48.375451 20952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:48.375525 20952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:48.375552 20952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:48.375572 20952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:48.375592 20952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:48.381098 20952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:48.382287 20952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:48.383378 20952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:48.391788 20960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:48.391796 20957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:48.391813 20958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:48.392140 20952 server_base.cc:1034] running on GCE node
I20260926 14:48:48.392709 20952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:48.393791 20952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:48.394941 20952 hybrid_clock.cc:648] HybridClock initialized: now 1790434128394927 us; error 31 us; skew 500 ppm
Sep 26 14:48:48 dist-test-slave-s7k6 krb5kdc[20939](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:48.405562 20952 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:48.406571 20952 webserver.cc:460] Webserver started at http://127.0.0.1:34067/ using document root <none> and password file <none>
I20260926 14:48:48.407243 20952 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:48.407307 20952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:48.407501 20952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:48.409613 20952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "fb1ee5ac65094235ac783bed3c72dd98"
format_stamp: "Formatted at 2026-09-26 14:48:48 on dist-test-slave-s7k6"
server_key: "ef054d7d876816ffae43269002e55c10"
server_key_iv: "fe76b45a2cca5588a0ea9503a269d694"
server_key_version: "encryptionkey@0"
I20260926 14:48:48.410187 20952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb1ee5ac65094235ac783bed3c72dd98"
format_stamp: "Formatted at 2026-09-26 14:48:48 on dist-test-slave-s7k6"
server_key: "ef054d7d876816ffae43269002e55c10"
server_key_iv: "fe76b45a2cca5588a0ea9503a269d694"
server_key_version: "encryptionkey@0"
I20260926 14:48:48.414085 20952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:48:48.416592 20967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:48.418002 20952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 14:48:48.418131 20952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "fb1ee5ac65094235ac783bed3c72dd98"
format_stamp: "Formatted at 2026-09-26 14:48:48 on dist-test-slave-s7k6"
server_key: "ef054d7d876816ffae43269002e55c10"
server_key_iv: "fe76b45a2cca5588a0ea9503a269d694"
server_key_version: "encryptionkey@0"
I20260926 14:48:48.418242 20952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:48.436053 20952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:48.439543 20952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:48.439741 20952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:48.446832 20952 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46497
I20260926 14:48:48.446866 21019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46497 every 8 connection(s)
I20260926 14:48:48.447824 20952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:48.450729 21020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:48.453555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 20952
I20260926 14:48:48.453691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:48.453943 392 external_mini_cluster.cc:1428] Setting key c52f6757ad423cd584690cba28cf763a
I20260926 14:48:48.456682 21020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98: Bootstrap starting.
I20260926 14:48:48.458956 21020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:48.459735 21020 log.cc:824] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:48.461722 21020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98: No bootstrap required, opened a new log
Sep 26 14:48:48 dist-test-slave-s7k6 krb5kdc[20939](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:48:48.465281 21020 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1ee5ac65094235ac783bed3c72dd98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260926 14:48:48.465520 21020 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:48.465556 21020 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1ee5ac65094235ac783bed3c72dd98, State: Initialized, Role: FOLLOWER
I20260926 14:48:48.466013 21020 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1ee5ac65094235ac783bed3c72dd98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260926 14:48:48.466140 21020 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:48.466197 21020 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:48.466287 21020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:48.467223 21020 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1ee5ac65094235ac783bed3c72dd98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260926 14:48:48.467571 21020 leader_election.cc:302] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb1ee5ac65094235ac783bed3c72dd98; no voters:
I20260926 14:48:48.467888 21020 leader_election.cc:288] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:48.467988 21025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:48.468154 21025 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [term 1 LEADER]: Becoming Leader. State: Replica: fb1ee5ac65094235ac783bed3c72dd98, State: Running, Role: LEADER
I20260926 14:48:48.468516 21025 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1ee5ac65094235ac783bed3c72dd98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } }
I20260926 14:48:48.468714 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:48.455306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60589 (local address 127.0.0.1:46497)
0926 14:48:48.455759 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:48.455778 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:48.455837 (+ 59us) server_negotiation.cc:407] Connection header received
0926 14:48:48.456461 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:48.456478 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:48.456874 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:48.457197 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:48.458038 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:48.459109 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:48.460099 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:48.460580 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:48.460727 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:48.463402 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:48.463407 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:48.463421 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:48.463448 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:48.465575 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:48.466010 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:48.466015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:48.466019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:48.466141 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:48.466422 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:48.466426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:48.466428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:48.466926 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:48.467100 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:48.467378 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 14:48:48.467665 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":144,"threads_started":1}
I20260926 14:48:48.469201 21020 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:48.469736 21025 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb1ee5ac65094235ac783bed3c72dd98. Latest consensus state: current_term: 1 leader_uuid: "fb1ee5ac65094235ac783bed3c72dd98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1ee5ac65094235ac783bed3c72dd98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } } }
I20260926 14:48:48.469851 21025 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:48.470256 21026 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb1ee5ac65094235ac783bed3c72dd98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb1ee5ac65094235ac783bed3c72dd98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46497 } } }
I20260926 14:48:48.470412 21026 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:48.471004 21030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:48.471984 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:48.474363 21030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:48.480482 21030 catalog_manager.cc:1347] Generated new cluster ID: ce90a3a7dc3144d8815e1e216f250a57
I20260926 14:48:48.480600 21030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:48.481590 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:48.473811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50141 (local address 127.0.0.1:46497)
0926 14:48:48.473923 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:48.473927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:48.474128 (+ 201us) server_negotiation.cc:407] Connection header received
0926 14:48:48.474242 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:48.474245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:48.474284 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:48.474342 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:48.475022 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:48.475527 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:48.476299 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:48.476543 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:48.476629 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:48.478128 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:48.478131 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:48.478135 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:48.478183 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:48.479871 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:48.480367 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:48.480371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:48.480373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:48.480454 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:48.480746 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:48.480749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:48.480751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:48.480967 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:48.481093 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:48.481342 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 14:48:48.481452 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:48:48.488405 21030 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:48.489625 21030 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:48.501076 21030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb1ee5ac65094235ac783bed3c72dd98: Generated new TSK 0
I20260926 14:48:48.501864 21030 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:48.509747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 20952
2026-09-26T14:48:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:48 dist-test-slave-s7k6 krb5kdc[21051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:48 dist-test-slave-s7k6 krb5kdc[21051](info): set up 2 sockets
Sep 26 14:48:48 dist-test-slave-s7k6 krb5kdc[21051](info): commencing operation
krb5kdc: starting...
W20260926 14:48:49.454651 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:49 dist-test-slave-s7k6 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:49.571923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35543
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:49.681864 21064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:49.682154 21064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:49.682195 21064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:49.686236 21064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:49.686298 21064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:49.686316 21064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:49.686331 21064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:49.686345 21064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:49.691658 21064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:49.692656 21064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:49.693698 21064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:48:49.702489 21064 server_base.cc:1034] running on GCE node
W20260926 14:48:49.702406 21069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:49.702396 21070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:49.702430 21072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:49.703295 21064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:49.704461 21064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:49.705646 21064 hybrid_clock.cc:648] HybridClock initialized: now 1790434129705636 us; error 31 us; skew 500 ppm
Sep 26 14:48:49 dist-test-slave-s7k6 krb5kdc[21051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434129, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:49.708510 21064 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:49.709424 21064 webserver.cc:460] Webserver started at http://127.0.0.1:35807/ using document root <none> and password file <none>
I20260926 14:48:49.709972 21064 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:49.710038 21064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:49.710228 21064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:49.712314 21064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "eb63fa160c9c4201a2cf6bf9c000e645"
format_stamp: "Formatted at 2026-09-26 14:48:49 on dist-test-slave-s7k6"
server_key: "82f12700d4178e4e1553bde34c7009d8"
server_key_iv: "0141d977a29c68cb03edbe98a8373b52"
server_key_version: "encryptionkey@0"
I20260926 14:48:49.712847 21064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "eb63fa160c9c4201a2cf6bf9c000e645"
format_stamp: "Formatted at 2026-09-26 14:48:49 on dist-test-slave-s7k6"
server_key: "82f12700d4178e4e1553bde34c7009d8"
server_key_iv: "0141d977a29c68cb03edbe98a8373b52"
server_key_version: "encryptionkey@0"
I20260926 14:48:49.716500 21064 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260926 14:48:49.719028 21079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:49.720541 21064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:48:49.720642 21064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "eb63fa160c9c4201a2cf6bf9c000e645"
format_stamp: "Formatted at 2026-09-26 14:48:49 on dist-test-slave-s7k6"
server_key: "82f12700d4178e4e1553bde34c7009d8"
server_key_iv: "0141d977a29c68cb03edbe98a8373b52"
server_key_version: "encryptionkey@0"
I20260926 14:48:49.720736 21064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:49.751453 21064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:49.754658 21064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:49.754854 21064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:49.761595 21064 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35543
I20260926 14:48:49.761665 21131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35543 every 8 connection(s)
I20260926 14:48:49.762534 21064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:49.765393 21132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:49.768057 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 21064
I20260926 14:48:49.768185 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:49.768425 392 external_mini_cluster.cc:1428] Setting key a8db0d2afe3da4643f7997c9665a23f2
I20260926 14:48:49.771581 21132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645: Bootstrap starting.
I20260926 14:48:49.774227 21132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:49.775089 21132 log.cc:824] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:49.777287 21132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645: No bootstrap required, opened a new log
I20260926 14:48:49.778451 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:49.769701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50023 (local address 127.0.0.1:35543)
0926 14:48:49.770138 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:49.770157 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:49.770200 (+ 43us) server_negotiation.cc:407] Connection header received
0926 14:48:49.770997 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:49.771021 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:49.771446 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:49.771817 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:49.772231 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:49.773688 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:49.774640 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:49.775366 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:49.775547 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:49.776396 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:49.776402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:49.776417 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:49.776913 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:49.776984 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:49.777015 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:49.777123 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:49.777265 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 14:48:49.777491 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":110,"threads_started":1}
I20260926 14:48:49.781145 21132 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63fa160c9c4201a2cf6bf9c000e645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35543 } }
I20260926 14:48:49.781387 21132 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:49.781445 21132 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb63fa160c9c4201a2cf6bf9c000e645, State: Initialized, Role: FOLLOWER
I20260926 14:48:49.781505 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:49.781888 21132 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63fa160c9c4201a2cf6bf9c000e645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35543 } }
I20260926 14:48:49.782039 21132 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:49.782102 21132 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:49.782198 21132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:49.783219 21132 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63fa160c9c4201a2cf6bf9c000e645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35543 } }
I20260926 14:48:49.783576 21132 leader_election.cc:302] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb63fa160c9c4201a2cf6bf9c000e645; no voters:
I20260926 14:48:49.783895 21132 leader_election.cc:288] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:49.783974 21142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:49.784168 21142 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [term 1 LEADER]: Becoming Leader. State: Replica: eb63fa160c9c4201a2cf6bf9c000e645, State: Running, Role: LEADER
I20260926 14:48:49.784516 21142 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63fa160c9c4201a2cf6bf9c000e645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35543 } }
I20260926 14:48:49.785228 21132 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:49.786170 21143 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb63fa160c9c4201a2cf6bf9c000e645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63fa160c9c4201a2cf6bf9c000e645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35543 } } }
I20260926 14:48:49.786296 21143 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:49.786537 21144 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb63fa160c9c4201a2cf6bf9c000e645. Latest consensus state: current_term: 1 leader_uuid: "eb63fa160c9c4201a2cf6bf9c000e645" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb63fa160c9c4201a2cf6bf9c000e645" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35543 } } }
I20260926 14:48:49.786612 21144 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:49.787550 21151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:49.788276 21135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:49.783625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37241 (local address 127.0.0.1:35543)
0926 14:48:49.783771 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:49.783776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:49.784056 (+ 280us) server_negotiation.cc:407] Connection header received
0926 14:48:49.784213 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:49.784217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:49.784271 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:49.784339 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:49.784710 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:49.785454 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:49.786650 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:49.787011 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:49.787125 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:49.787856 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:49.787860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:49.787862 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:49.787874 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:49.787902 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:49.787913 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:49.787988 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:49.788095 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 14:48:49.788152 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:48:49.791287 21151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:49.797209 21151 catalog_manager.cc:1347] Generated new cluster ID: 4804748462754cf5b06f58552526ef58
I20260926 14:48:49.797279 21151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:49.822837 21151 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:49.824200 21151 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:49.829618 21151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb63fa160c9c4201a2cf6bf9c000e645: Generated new TSK 0
I20260926 14:48:49.830446 21151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:49.838867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 21064
2026-09-26T14:48:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:49 dist-test-slave-s7k6 krb5kdc[21163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:49 dist-test-slave-s7k6 krb5kdc[21163](info): set up 2 sockets
Sep 26 14:48:49 dist-test-slave-s7k6 krb5kdc[21163](info): commencing operation
krb5kdc: starting...
W20260926 14:48:50.802898 392 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:50 dist-test-slave-s7k6 krb5kdc[21163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:50.918020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:51.030795 21176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:51.031163 21176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:51.031224 21176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:51.035360 21176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:51.035437 21176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:51.035465 21176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:51.035485 21176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:51.035511 21176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:51.041059 21176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41755
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:51.042194 21176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:51.043288 21176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:51.051550 21181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:51.051550 21182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:51.051553 21184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:51.052037 21176 server_base.cc:1034] running on GCE node
I20260926 14:48:51.052614 21176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:51.053694 21176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:51.054818 21176 hybrid_clock.cc:648] HybridClock initialized: now 1790434131054795 us; error 32 us; skew 500 ppm
Sep 26 14:48:51 dist-test-slave-s7k6 krb5kdc[21163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434131, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:51.057721 21176 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:51.058724 21176 webserver.cc:460] Webserver started at http://127.0.0.1:41307/ using document root <none> and password file <none>
I20260926 14:48:51.059329 21176 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:51.059404 21176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:51.059612 21176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:51.061920 21176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "cc3067b747304c74b76efb2b5440ee63"
format_stamp: "Formatted at 2026-09-26 14:48:51 on dist-test-slave-s7k6"
server_key: "f1da2b4c0f7d9172e38013c1c76ebe42"
server_key_iv: "b6618247f0f9f1504abebc9a5957ecd5"
server_key_version: "encryptionkey@0"
I20260926 14:48:51.062496 21176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "cc3067b747304c74b76efb2b5440ee63"
format_stamp: "Formatted at 2026-09-26 14:48:51 on dist-test-slave-s7k6"
server_key: "f1da2b4c0f7d9172e38013c1c76ebe42"
server_key_iv: "b6618247f0f9f1504abebc9a5957ecd5"
server_key_version: "encryptionkey@0"
I20260926 14:48:51.066376 21176 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260926 14:48:51.068957 21191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:51.070340 21176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:48:51.070506 21176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "cc3067b747304c74b76efb2b5440ee63"
format_stamp: "Formatted at 2026-09-26 14:48:51 on dist-test-slave-s7k6"
server_key: "f1da2b4c0f7d9172e38013c1c76ebe42"
server_key_iv: "b6618247f0f9f1504abebc9a5957ecd5"
server_key_version: "encryptionkey@0"
I20260926 14:48:51.070612 21176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:51.075232 21176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:51.078128 21176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:51.078289 21176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:51.085049 21176 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41755
I20260926 14:48:51.085059 21243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41755 every 8 connection(s)
I20260926 14:48:51.085978 21176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:51.088735 21244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:51.094002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 21176
I20260926 14:48:51.094131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:51.094254 21244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63: Bootstrap starting.
I20260926 14:48:51.094504 392 external_mini_cluster.cc:1428] Setting key dbf001662557bb58c9aa39ebed449468
I20260926 14:48:51.096946 21244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:51.097707 21244 log.cc:824] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:51.099931 21244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63: No bootstrap required, opened a new log
I20260926 14:48:51.100081 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:51.095877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59269 (local address 127.0.0.1:41755)
0926 14:48:51.096444 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:51.096465 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:51.096514 (+ 49us) server_negotiation.cc:407] Connection header received
0926 14:48:51.097247 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:51.097267 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:51.097600 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:51.097947 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:51.098705 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:51.098710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:51.098723 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:51.098732 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:51.098818 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:51.098844 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:51.098937 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:51.099045 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 14:48:51.099198 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":135,"threads_started":1}
I20260926 14:48:51.103108 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:51.104210 21244 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc3067b747304c74b76efb2b5440ee63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } }
I20260926 14:48:51.104514 21244 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:51.104581 21244 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc3067b747304c74b76efb2b5440ee63, State: Initialized, Role: FOLLOWER
I20260926 14:48:51.105075 21244 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc3067b747304c74b76efb2b5440ee63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } }
I20260926 14:48:51.105226 21244 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:51.105340 21244 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:51.105449 21244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:51.106556 21244 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc3067b747304c74b76efb2b5440ee63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } }
I20260926 14:48:51.106952 21244 leader_election.cc:302] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc3067b747304c74b76efb2b5440ee63; no voters:
I20260926 14:48:51.107062 21247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:51.105377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47293 (local address 127.0.0.1:41755)
0926 14:48:51.105500 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:51.105505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:51.105743 (+ 238us) server_negotiation.cc:407] Connection header received
0926 14:48:51.105825 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:51.105828 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:51.105863 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:51.105940 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:51.106630 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:51.106634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:51.106636 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:51.106648 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:51.106675 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:51.106687 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:51.106790 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:51.106871 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 14:48:51.106907 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:48:51.107295 21244 leader_election.cc:288] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:51.107362 21254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:51.107549 21254 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [term 1 LEADER]: Becoming Leader. State: Replica: cc3067b747304c74b76efb2b5440ee63, State: Running, Role: LEADER
I20260926 14:48:51.107848 21254 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc3067b747304c74b76efb2b5440ee63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } }
I20260926 14:48:51.108604 21244 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:51.109649 21256 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc3067b747304c74b76efb2b5440ee63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc3067b747304c74b76efb2b5440ee63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } } }
I20260926 14:48:51.109771 21256 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:51.110054 21255 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc3067b747304c74b76efb2b5440ee63. Latest consensus state: current_term: 1 leader_uuid: "cc3067b747304c74b76efb2b5440ee63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc3067b747304c74b76efb2b5440ee63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41755 } } }
I20260926 14:48:51.110139 21263 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:51.110162 21255 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:51.112848 21263 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:51.118089 21263 catalog_manager.cc:1347] Generated new cluster ID: f3d9a666b9ed412984bf1f1e73e81c15
I20260926 14:48:51.118160 21263 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:51.150108 21263 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:51.150468 21263 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:51.161130 21263 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc3067b747304c74b76efb2b5440ee63: Generated new TSK 0
I20260926 14:48:51.161926 21263 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:51.174266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 21176
2026-09-26T14:48:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:51 dist-test-slave-s7k6 krb5kdc[21275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:51 dist-test-slave-s7k6 krb5kdc[21275](info): set up 2 sockets
Sep 26 14:48:51 dist-test-slave-s7k6 krb5kdc[21275](info): commencing operation
krb5kdc: starting...
W20260926 14:48:52.126812 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:52 dist-test-slave-s7k6 krb5kdc[21275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:52.242837 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36877
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:52.354640 21288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:52.354933 21288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:52.354981 21288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:52.359004 21288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:52.359064 21288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:52.359082 21288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:52.359098 21288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:52.359113 21288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:52.364511 21288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:52.365581 21288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:52.366611 21288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:52.374802 21293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:52.374797 21294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:52.374802 21296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:52.375118 21288 server_base.cc:1034] running on GCE node
I20260926 14:48:52.375818 21288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:52.376993 21288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:52.378170 21288 hybrid_clock.cc:648] HybridClock initialized: now 1790434132378135 us; error 48 us; skew 500 ppm
Sep 26 14:48:52 dist-test-slave-s7k6 krb5kdc[21275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:52.381031 21288 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:52.381981 21288 webserver.cc:460] Webserver started at http://127.0.0.1:39695/ using document root <none> and password file <none>
I20260926 14:48:52.382534 21288 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:52.382601 21288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:52.382839 21288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:52.384935 21288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "501f90d2e2524ca8b4f2057f20d563e1"
format_stamp: "Formatted at 2026-09-26 14:48:52 on dist-test-slave-s7k6"
server_key: "46bb8119a51a351217db97a916418b9d"
server_key_iv: "3e768ea542b4e90f6e5042dceb7369c5"
server_key_version: "encryptionkey@0"
I20260926 14:48:52.385481 21288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "501f90d2e2524ca8b4f2057f20d563e1"
format_stamp: "Formatted at 2026-09-26 14:48:52 on dist-test-slave-s7k6"
server_key: "46bb8119a51a351217db97a916418b9d"
server_key_iv: "3e768ea542b4e90f6e5042dceb7369c5"
server_key_version: "encryptionkey@0"
I20260926 14:48:52.389324 21288 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260926 14:48:52.391935 21303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:52.393251 21288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:52.393383 21288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "501f90d2e2524ca8b4f2057f20d563e1"
format_stamp: "Formatted at 2026-09-26 14:48:52 on dist-test-slave-s7k6"
server_key: "46bb8119a51a351217db97a916418b9d"
server_key_iv: "3e768ea542b4e90f6e5042dceb7369c5"
server_key_version: "encryptionkey@0"
I20260926 14:48:52.393491 21288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:52.413009 21288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:52.416131 21288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:52.416316 21288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:52.423472 21288 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36877
I20260926 14:48:52.423488 21355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36877 every 8 connection(s)
I20260926 14:48:52.424438 21288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:52.427340 21356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:52.428951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 21288
I20260926 14:48:52.429087 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:52.429337 392 external_mini_cluster.cc:1428] Setting key 6c91ab338f301f383df1bd833c6ba1b7
I20260926 14:48:52.433836 21356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1: Bootstrap starting.
I20260926 14:48:52.436429 21356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:52.437314 21356 log.cc:824] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1: Log is configured to *not* fsync() on all Append() calls
Sep 26 14:48:52 dist-test-slave-s7k6 krb5kdc[21275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:48:52.439514 21356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1: No bootstrap required, opened a new log
I20260926 14:48:52.444025 21356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f90d2e2524ca8b4f2057f20d563e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260926 14:48:52.444304 21356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:52.444361 21356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501f90d2e2524ca8b4f2057f20d563e1, State: Initialized, Role: FOLLOWER
I20260926 14:48:52.444725 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:52.430688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:34331 (local address 127.0.0.1:36877)
0926 14:48:52.431126 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:52.431145 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:52.431187 (+ 42us) server_negotiation.cc:407] Connection header received
0926 14:48:52.431982 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:52.432006 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:52.432471 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:52.432850 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:52.433538 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:52.434971 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:52.435865 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:52.436549 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:52.436716 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:52.439500 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:52.439506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:52.439523 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:52.439556 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:52.441446 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:52.441980 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:52.441987 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:52.441993 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:52.442137 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:52.442399 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:52.442402 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:52.442403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:52.442940 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:52.443164 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:52.443444 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 14:48:52.443675 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":114,"threads_started":1}
I20260926 14:48:52.444870 21356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f90d2e2524ca8b4f2057f20d563e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260926 14:48:52.445029 21356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:52.445096 21356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:52.445184 21356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:52.446261 21356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f90d2e2524ca8b4f2057f20d563e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260926 14:48:52.446623 21356 leader_election.cc:302] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 501f90d2e2524ca8b4f2057f20d563e1; no voters:
I20260926 14:48:52.446985 21356 leader_election.cc:288] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:52.447247 21361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:52.447491 21361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [term 1 LEADER]: Becoming Leader. State: Replica: 501f90d2e2524ca8b4f2057f20d563e1, State: Running, Role: LEADER
I20260926 14:48:52.447803 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:52.447862 21361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f90d2e2524ca8b4f2057f20d563e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } }
I20260926 14:48:52.448266 21356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:52.449556 21362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "501f90d2e2524ca8b4f2057f20d563e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f90d2e2524ca8b4f2057f20d563e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } } }
I20260926 14:48:52.449697 21362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:52.449996 21363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 501f90d2e2524ca8b4f2057f20d563e1. Latest consensus state: current_term: 1 leader_uuid: "501f90d2e2524ca8b4f2057f20d563e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f90d2e2524ca8b4f2057f20d563e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36877 } } }
I20260926 14:48:52.450484 21363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:52.450623 21379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:52.453926 21379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:52.458910 21359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:52.450104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:57941 (local address 127.0.0.1:36877)
0926 14:48:52.450208 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:52.450212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:52.450712 (+ 500us) server_negotiation.cc:407] Connection header received
0926 14:48:52.450852 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:52.450855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:52.450903 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:52.450985 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:52.451775 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:52.452556 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:52.453626 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:52.453849 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:52.453926 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:52.455538 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:52.455542 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:52.455545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:52.455570 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:52.457174 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:52.457632 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:52.457636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:52.457638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:52.457727 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:52.458014 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:52.458017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:52.458019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:52.458238 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:52.458400 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:52.458590 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 14:48:52.458697 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 14:48:52.463181 21379 catalog_manager.cc:1347] Generated new cluster ID: 682d1e3c8bdc4a0296dcdb24d43e615f
I20260926 14:48:52.463258 21379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:52.475807 21379 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:52.477015 21379 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:52.482649 21379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 501f90d2e2524ca8b4f2057f20d563e1: Generated new TSK 0
I20260926 14:48:52.483355 21379 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:52.494148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 21288
W20260926 14:48:52.498903 21273 connection.cc:535] client connection to 127.0.0.1:36877 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36877): Connection reset by peer (error 104)
2026-09-26T14:48:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:52 dist-test-slave-s7k6 krb5kdc[21387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:52 dist-test-slave-s7k6 krb5kdc[21387](info): set up 2 sockets
Sep 26 14:48:52 dist-test-slave-s7k6 krb5kdc[21387](info): commencing operation
krb5kdc: starting...
W20260926 14:48:53.438735 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:53 dist-test-slave-s7k6 krb5kdc[21387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:53.553934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:53.666657 21400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:53.666970 21400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:53.667032 21400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:53.671206 21400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:53.671268 21400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:53.671294 21400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:53.671321 21400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:53.671350 21400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:53.676719 21400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:53.677757 21400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:53.678776 21400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:53.687083 21406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:53.687094 21405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:53.687122 21408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:53.687299 21400 server_base.cc:1034] running on GCE node
I20260926 14:48:53.687896 21400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:53.688984 21400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:53.690140 21400 hybrid_clock.cc:648] HybridClock initialized: now 1790434133690124 us; error 34 us; skew 500 ppm
Sep 26 14:48:53 dist-test-slave-s7k6 krb5kdc[21387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434133, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:53.697440 21400 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:53.698421 21400 webserver.cc:460] Webserver started at http://127.0.0.1:41945/ using document root <none> and password file <none>
I20260926 14:48:53.699033 21400 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:53.699096 21400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:53.699283 21400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:53.701390 21400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "fa2b7bd6d16249a8bbbe6f523c9c6388"
format_stamp: "Formatted at 2026-09-26 14:48:53 on dist-test-slave-s7k6"
server_key: "514e5e83adb48ee0823600d1184ce5e5"
server_key_iv: "c948590ec0ff064bbc432a25db2d1b74"
server_key_version: "encryptionkey@0"
I20260926 14:48:53.701996 21400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "fa2b7bd6d16249a8bbbe6f523c9c6388"
format_stamp: "Formatted at 2026-09-26 14:48:53 on dist-test-slave-s7k6"
server_key: "514e5e83adb48ee0823600d1184ce5e5"
server_key_iv: "c948590ec0ff064bbc432a25db2d1b74"
server_key_version: "encryptionkey@0"
I20260926 14:48:53.705797 21400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 14:48:53.708345 21415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:53.709733 21400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:48:53.709849 21400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "fa2b7bd6d16249a8bbbe6f523c9c6388"
format_stamp: "Formatted at 2026-09-26 14:48:53 on dist-test-slave-s7k6"
server_key: "514e5e83adb48ee0823600d1184ce5e5"
server_key_iv: "c948590ec0ff064bbc432a25db2d1b74"
server_key_version: "encryptionkey@0"
I20260926 14:48:53.709945 21400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:53.729393 21400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:53.732563 21400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:53.732721 21400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:53.739647 21400 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41057
I20260926 14:48:53.739676 21467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41057 every 8 connection(s)
I20260926 14:48:53.740665 21400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:53.743568 21468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:53.749220 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388: Bootstrap starting.
I20260926 14:48:53.750272 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 21400
I20260926 14:48:53.750379 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:53.750612 392 external_mini_cluster.cc:1428] Setting key 7b6474a9879ea4caa81c2afb3266cfcf
I20260926 14:48:53.752000 21468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:53.752727 21468 log.cc:824] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:53.754887 21468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388: No bootstrap required, opened a new log
I20260926 14:48:53.759008 21468 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2b7bd6d16249a8bbbe6f523c9c6388" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260926 14:48:53.759248 21468 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:53.759315 21468 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa2b7bd6d16249a8bbbe6f523c9c6388, State: Initialized, Role: FOLLOWER
I20260926 14:48:53.759505 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:53.751749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:54333 (local address 127.0.0.1:41057)
0926 14:48:53.752185 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:53.752201 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:53.752237 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:48:53.752876 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:53.752898 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:53.753359 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:53.753679 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:53.754027 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:53.755050 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:53.756214 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:53.756673 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:53.756790 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:53.757674 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:53.757679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:53.757693 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:53.758080 (+ 387us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:53.758135 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:53.758169 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:53.758267 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:53.758453 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 14:48:53.758595 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":114,"threads_started":1}
I20260926 14:48:53.759790 21468 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2b7bd6d16249a8bbbe6f523c9c6388" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260926 14:48:53.759923 21468 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:53.759986 21468 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:53.760198 21468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:53.761191 21468 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2b7bd6d16249a8bbbe6f523c9c6388" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260926 14:48:53.761541 21468 leader_election.cc:302] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa2b7bd6d16249a8bbbe6f523c9c6388; no voters:
I20260926 14:48:53.761897 21468 leader_election.cc:288] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:53.761987 21473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:53.762202 21473 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [term 1 LEADER]: Becoming Leader. State: Replica: fa2b7bd6d16249a8bbbe6f523c9c6388, State: Running, Role: LEADER
I20260926 14:48:53.762456 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:53.762519 21473 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2b7bd6d16249a8bbbe6f523c9c6388" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260926 14:48:53.763311 21468 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:53.764008 21475 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa2b7bd6d16249a8bbbe6f523c9c6388. Latest consensus state: current_term: 1 leader_uuid: "fa2b7bd6d16249a8bbbe6f523c9c6388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2b7bd6d16249a8bbbe6f523c9c6388" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } } }
I20260926 14:48:53.764133 21475 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:53.764434 21474 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa2b7bd6d16249a8bbbe6f523c9c6388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa2b7bd6d16249a8bbbe6f523c9c6388" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } } }
I20260926 14:48:53.764531 21474 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388 [sys.catalog]: This master's current role is: LEADER
W20260926 14:48:53.768242 21493 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:48:53.768376 21493 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:48:53.768491 21486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:53.769490 21486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:53.769794 21471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:53.765875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:60939 (local address 127.0.0.1:41057)
0926 14:48:53.765973 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:53.765978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:53.766162 (+ 184us) server_negotiation.cc:407] Connection header received
0926 14:48:53.766269 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:53.766272 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:53.766315 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:53.766379 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:53.766626 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:53.767202 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:53.767998 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:53.768236 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:53.768318 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:53.769228 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:53.769232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:53.769235 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:53.769249 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:53.769275 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:53.769289 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:53.769350 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:53.769621 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 14:48:53.769668 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 14:48:53.775467 21486 catalog_manager.cc:1347] Generated new cluster ID: 897ff11ff7dd432f9235b86195c5ab10
I20260926 14:48:53.775544 21486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:53.786718 21486 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:53.787725 21486 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:53.800710 21486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa2b7bd6d16249a8bbbe6f523c9c6388: Generated new TSK 0
I20260926 14:48:53.801532 21486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:53.811617 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 21400
W20260926 14:48:53.816574 21385 connection.cc:535] client connection to 127.0.0.1:41057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41057): Connection reset by peer (error 104)
2026-09-26T14:48:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1317 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:53 dist-test-slave-s7k6 krb5kdc[21499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:53 dist-test-slave-s7k6 krb5kdc[21499](info): set up 2 sockets
Sep 26 14:48:53 dist-test-slave-s7k6 krb5kdc[21499](info): commencing operation
krb5kdc: starting...
W20260926 14:48:54.770838 392 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:54 dist-test-slave-s7k6 krb5kdc[21499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:54.885951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:54.997308 21512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:54.997704 21512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:54.997767 21512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:55.001864 21512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:55.001936 21512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:55.001963 21512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:55.001984 21512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:55.002007 21512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:55.007844 21512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:55.008965 21512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:55.010028 21512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:48:55.019212 21512 server_base.cc:1034] running on GCE node
W20260926 14:48:55.019163 21517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:55.019163 21518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:55.019168 21520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:55.020053 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:55.021253 21512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:55.022429 21512 hybrid_clock.cc:648] HybridClock initialized: now 1790434135022409 us; error 33 us; skew 500 ppm
Sep 26 14:48:55 dist-test-slave-s7k6 krb5kdc[21499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:55.025272 21512 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:55.026238 21512 webserver.cc:460] Webserver started at http://127.0.0.1:46671/ using document root <none> and password file <none>
I20260926 14:48:55.026852 21512 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:55.026928 21512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:55.027154 21512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:55.029374 21512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "f3464af892f14f68b05aae6567169fe3"
format_stamp: "Formatted at 2026-09-26 14:48:55 on dist-test-slave-s7k6"
server_key: "55627350f90c358f84673985bf5ba542"
server_key_iv: "cbf3dced6acb354d1f358e74f5f2e3b8"
server_key_version: "encryptionkey@0"
I20260926 14:48:55.029948 21512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3464af892f14f68b05aae6567169fe3"
format_stamp: "Formatted at 2026-09-26 14:48:55 on dist-test-slave-s7k6"
server_key: "55627350f90c358f84673985bf5ba542"
server_key_iv: "cbf3dced6acb354d1f358e74f5f2e3b8"
server_key_version: "encryptionkey@0"
I20260926 14:48:55.033824 21512 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260926 14:48:55.036420 21527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:55.037705 21512 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:48:55.037823 21512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "f3464af892f14f68b05aae6567169fe3"
format_stamp: "Formatted at 2026-09-26 14:48:55 on dist-test-slave-s7k6"
server_key: "55627350f90c358f84673985bf5ba542"
server_key_iv: "cbf3dced6acb354d1f358e74f5f2e3b8"
server_key_version: "encryptionkey@0"
I20260926 14:48:55.037925 21512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:55.042472 21512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:55.045300 21512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:55.045463 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:55.052198 21512 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46277
I20260926 14:48:55.052212 21579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46277 every 8 connection(s)
I20260926 14:48:55.053131 21512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:55.055989 21580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:55.061764 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3: Bootstrap starting.
I20260926 14:48:55.061821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 21512
I20260926 14:48:55.061952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:55.062197 392 external_mini_cluster.cc:1428] Setting key 7f48597ad3261fa5ae4d13af95718f68
I20260926 14:48:55.064426 21580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:55.065187 21580 log.cc:824] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:55.067297 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3: No bootstrap required, opened a new log
I20260926 14:48:55.068399 21583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:55.063581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:44771 (local address 127.0.0.1:46277)
0926 14:48:55.064017 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:55.064040 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:55.064078 (+ 38us) server_negotiation.cc:407] Connection header received
0926 14:48:55.064768 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:55.064788 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:55.065135 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:55.065483 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:55.067105 (+ 1622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:55.067110 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:55.067122 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:55.067132 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:55.067184 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:55.067203 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:55.067292 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:55.067445 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 14:48:55.067616 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":114,"threads_started":1}
I20260926 14:48:55.070974 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:55.070971 21580 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3464af892f14f68b05aae6567169fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46277 } }
I20260926 14:48:55.071254 21580 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:55.071305 21580 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3464af892f14f68b05aae6567169fe3, State: Initialized, Role: FOLLOWER
I20260926 14:48:55.071770 21580 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3464af892f14f68b05aae6567169fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46277 } }
I20260926 14:48:55.071928 21580 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:55.072012 21580 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:55.072108 21580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:55.073228 21580 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3464af892f14f68b05aae6567169fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46277 } }
I20260926 14:48:55.073767 21580 leader_election.cc:302] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3464af892f14f68b05aae6567169fe3; no voters:
I20260926 14:48:55.074120 21580 leader_election.cc:288] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:55.074218 21590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:55.074436 21590 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [term 1 LEADER]: Becoming Leader. State: Replica: f3464af892f14f68b05aae6567169fe3, State: Running, Role: LEADER
I20260926 14:48:55.075016 21583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:55.073199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:55765 (local address 127.0.0.1:46277)
0926 14:48:55.073321 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:55.073325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:55.073631 (+ 306us) server_negotiation.cc:407] Connection header received
0926 14:48:55.073673 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:55.073675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:55.073708 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:55.073768 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:55.074572 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:55.074575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:55.074578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:55.074590 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:55.074618 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:55.074631 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:55.074684 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:55.074859 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 14:48:55.074902 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 14:48:55.074836 21590 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3464af892f14f68b05aae6567169fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46277 } }
I20260926 14:48:55.075479 21580 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:55.076727 21591 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3464af892f14f68b05aae6567169fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3464af892f14f68b05aae6567169fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46277 } } }
I20260926 14:48:55.076718 21592 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3464af892f14f68b05aae6567169fe3. Latest consensus state: current_term: 1 leader_uuid: "f3464af892f14f68b05aae6567169fe3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3464af892f14f68b05aae6567169fe3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46277 } } }
I20260926 14:48:55.076853 21592 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:55.076853 21591 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:55.077304 21601 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:55.080547 21601 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:55.086274 21601 catalog_manager.cc:1347] Generated new cluster ID: e65882e56eff4af8948bf0d99a452ebd
I20260926 14:48:55.086344 21601 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:55.101135 21601 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:55.101522 21601 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:55.112172 21601 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3464af892f14f68b05aae6567169fe3: Generated new TSK 0
I20260926 14:48:55.113051 21601 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:55.126129 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 21512
2026-09-26T14:48:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:55 dist-test-slave-s7k6 krb5kdc[21611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:55 dist-test-slave-s7k6 krb5kdc[21611](info): set up 2 sockets
Sep 26 14:48:55 dist-test-slave-s7k6 krb5kdc[21611](info): commencing operation
krb5kdc: starting...
W20260926 14:48:56.074952 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:56 dist-test-slave-s7k6 krb5kdc[21611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:56.194005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:56.307075 21624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:56.307376 21624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:56.307441 21624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:56.311579 21624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:56.311651 21624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:56.311676 21624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:56.311699 21624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:56.311728 21624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:56.317139 21624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43581
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:56.318246 21624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:56.319345 21624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:56.328035 21630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:56.328092 21624 server_base.cc:1034] running on GCE node
W20260926 14:48:56.328048 21632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:56.328028 21629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:56.328862 21624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:56.330080 21624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:56.331265 21624 hybrid_clock.cc:648] HybridClock initialized: now 1790434136331243 us; error 40 us; skew 500 ppm
Sep 26 14:48:56 dist-test-slave-s7k6 krb5kdc[21611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:56.334226 21624 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:56.335222 21624 webserver.cc:460] Webserver started at http://127.0.0.1:39871/ using document root <none> and password file <none>
I20260926 14:48:56.335829 21624 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:56.335902 21624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:56.336118 21624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:56.338312 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "1dde8e946f4945e6a340e43ab5e96210"
format_stamp: "Formatted at 2026-09-26 14:48:56 on dist-test-slave-s7k6"
server_key: "7fe1cf74452fa8136956a3a8014d2d04"
server_key_iv: "53bbdfd5f078996a3ab70c29fdb78e9d"
server_key_version: "encryptionkey@0"
I20260926 14:48:56.338954 21624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "1dde8e946f4945e6a340e43ab5e96210"
format_stamp: "Formatted at 2026-09-26 14:48:56 on dist-test-slave-s7k6"
server_key: "7fe1cf74452fa8136956a3a8014d2d04"
server_key_iv: "53bbdfd5f078996a3ab70c29fdb78e9d"
server_key_version: "encryptionkey@0"
I20260926 14:48:56.343703 21624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:48:56.346846 21639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:56.348474 21624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:56.348615 21624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "1dde8e946f4945e6a340e43ab5e96210"
format_stamp: "Formatted at 2026-09-26 14:48:56 on dist-test-slave-s7k6"
server_key: "7fe1cf74452fa8136956a3a8014d2d04"
server_key_iv: "53bbdfd5f078996a3ab70c29fdb78e9d"
server_key_version: "encryptionkey@0"
I20260926 14:48:56.348798 21624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:56.365379 21624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:56.369321 21624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:56.369525 21624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:56.377851 21624 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43581
I20260926 14:48:56.377893 21691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43581 every 8 connection(s)
I20260926 14:48:56.379186 21624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:56.380249 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 21624
I20260926 14:48:56.380390 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:56.380694 392 external_mini_cluster.cc:1428] Setting key 55cbe55e6f058239437c89822b67072e
I20260926 14:48:56.383090 21692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 26 14:48:56 dist-test-slave-s7k6 krb5kdc[21611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 14:48:56.389835 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210: Bootstrap starting.
I20260926 14:48:56.392280 21692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:56.393011 21692 log.cc:824] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:56.395046 21692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210: No bootstrap required, opened a new log
I20260926 14:48:56.395108 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:56.382074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:40521 (local address 127.0.0.1:43581)
0926 14:48:56.382537 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:56.382556 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:56.382599 (+ 43us) server_negotiation.cc:407] Connection header received
0926 14:48:56.383424 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:56.383446 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:56.383841 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:56.384169 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:56.385009 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:56.386038 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:56.386944 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:56.387441 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:56.387588 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:56.390449 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:56.390455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:56.390470 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:56.390500 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:56.392169 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:56.392710 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:56.392715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:56.392720 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:56.392832 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:56.393148 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:56.393152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:56.393153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:56.393557 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:56.393757 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:56.394084 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 14:48:56.394307 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":163,"threads_started":1}
I20260926 14:48:56.398289 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:56.398993 21692 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dde8e946f4945e6a340e43ab5e96210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } }
I20260926 14:48:56.399228 21692 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:56.399286 21692 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dde8e946f4945e6a340e43ab5e96210, State: Initialized, Role: FOLLOWER
I20260926 14:48:56.399788 21692 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dde8e946f4945e6a340e43ab5e96210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } }
I20260926 14:48:56.399922 21692 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:56.399981 21692 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:56.400076 21692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:56.401180 21692 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dde8e946f4945e6a340e43ab5e96210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } }
I20260926 14:48:56.401539 21692 leader_election.cc:302] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dde8e946f4945e6a340e43ab5e96210; no voters:
I20260926 14:48:56.401834 21692 leader_election.cc:288] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:56.401990 21702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:56.402204 21702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [term 1 LEADER]: Becoming Leader. State: Replica: 1dde8e946f4945e6a340e43ab5e96210, State: Running, Role: LEADER
I20260926 14:48:56.402558 21702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dde8e946f4945e6a340e43ab5e96210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } }
I20260926 14:48:56.403226 21692 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:56.404296 21704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dde8e946f4945e6a340e43ab5e96210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dde8e946f4945e6a340e43ab5e96210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } } }
I20260926 14:48:56.404421 21704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:56.404700 21703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dde8e946f4945e6a340e43ab5e96210. Latest consensus state: current_term: 1 leader_uuid: "1dde8e946f4945e6a340e43ab5e96210" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dde8e946f4945e6a340e43ab5e96210" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43581 } } }
I20260926 14:48:56.404805 21703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:56.404886 21711 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:56.408249 21711 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:56.409715 21695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:56.400296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:44037 (local address 127.0.0.1:43581)
0926 14:48:56.400424 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:56.400428 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:56.400578 (+ 150us) server_negotiation.cc:407] Connection header received
0926 14:48:56.400673 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:56.400676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:56.400724 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:56.400802 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:56.401440 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:56.401971 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:56.402943 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:56.403206 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:56.403267 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:56.406271 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:56.406275 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:56.406279 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:48:56.406304 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:56.407862 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:56.408397 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:56.408401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:56.408403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:56.408476 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:48:56.408819 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:48:56.408823 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:48:56.408825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:48:56.409012 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:56.409146 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:56.409485 (+ 339us) server_negotiation.cc:299] Negotiation successful
0926 14:48:56.409570 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 14:48:56.414700 21711 catalog_manager.cc:1347] Generated new cluster ID: d31b1045f0f5493784eb45880df631cf
I20260926 14:48:56.414799 21711 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:56.423688 21711 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:56.425045 21711 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:56.439129 21711 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dde8e946f4945e6a340e43ab5e96210: Generated new TSK 0
I20260926 14:48:56.439934 21711 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:56.444433 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 21624
W20260926 14:48:56.449189 21609 connection.cc:535] client connection to 127.0.0.1:43581 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43581): Connection reset by peer (error 104)
2026-09-26T14:48:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:56 dist-test-slave-s7k6 krb5kdc[21723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:56 dist-test-slave-s7k6 krb5kdc[21723](info): set up 2 sockets
Sep 26 14:48:56 dist-test-slave-s7k6 krb5kdc[21723](info): commencing operation
krb5kdc: starting...
W20260926 14:48:57.398972 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:57 dist-test-slave-s7k6 krb5kdc[21723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:57.517719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:57.630133 21736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:57.630425 21736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:57.630468 21736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:57.634696 21736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:57.634802 21736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:57.634824 21736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:57.634840 21736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:57.634877 21736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:57.640535 21736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38305
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:57.641645 21736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:57.642854 21736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:48:57.651302 21736 server_base.cc:1034] running on GCE node
W20260926 14:48:57.651249 21741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:57.651250 21742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:57.651249 21744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:57.652009 21736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:57.653154 21736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:57.654327 21736 hybrid_clock.cc:648] HybridClock initialized: now 1790434137654307 us; error 37 us; skew 500 ppm
Sep 26 14:48:57 dist-test-slave-s7k6 krb5kdc[21723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:57.657241 21736 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:57.658243 21736 webserver.cc:460] Webserver started at http://127.0.0.1:41905/ using document root <none> and password file <none>
I20260926 14:48:57.658841 21736 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:57.658910 21736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:57.659096 21736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:57.661197 21736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "8d4c5017fe204212802cbc2df5a8a487"
format_stamp: "Formatted at 2026-09-26 14:48:57 on dist-test-slave-s7k6"
server_key: "180be06a3f8c94873cbdde7832efab9b"
server_key_iv: "048ead10e172548c71865e2a5f8b102d"
server_key_version: "encryptionkey@0"
I20260926 14:48:57.661742 21736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d4c5017fe204212802cbc2df5a8a487"
format_stamp: "Formatted at 2026-09-26 14:48:57 on dist-test-slave-s7k6"
server_key: "180be06a3f8c94873cbdde7832efab9b"
server_key_iv: "048ead10e172548c71865e2a5f8b102d"
server_key_version: "encryptionkey@0"
I20260926 14:48:57.666008 21736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 14:48:57.668845 21751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:57.670130 21736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 14:48:57.670291 21736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "8d4c5017fe204212802cbc2df5a8a487"
format_stamp: "Formatted at 2026-09-26 14:48:57 on dist-test-slave-s7k6"
server_key: "180be06a3f8c94873cbdde7832efab9b"
server_key_iv: "048ead10e172548c71865e2a5f8b102d"
server_key_version: "encryptionkey@0"
I20260926 14:48:57.670395 21736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:57.683601 21736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:57.686578 21736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:57.686733 21736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:57.693610 21736 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38305
I20260926 14:48:57.693634 21803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38305 every 8 connection(s)
I20260926 14:48:57.694633 21736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:57.697613 21804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:57.703756 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 21736
I20260926 14:48:57.703899 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:57.704175 392 external_mini_cluster.cc:1428] Setting key 3221ca4015a6bead1697f45218c581b1
I20260926 14:48:57.705569 21804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487: Bootstrap starting.
I20260926 14:48:57.708029 21804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:57.708781 21804 log.cc:824] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:57.711259 21804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487: No bootstrap required, opened a new log
I20260926 14:48:57.714689 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:57.705706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:43369 (local address 127.0.0.1:38305)
0926 14:48:57.706200 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:57.706218 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:57.706258 (+ 40us) server_negotiation.cc:407] Connection header received
0926 14:48:57.707011 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:57.707032 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:57.707447 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:57.707802 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:57.708269 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:57.709347 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:57.710344 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:57.711082 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:57.711257 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:57.712181 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:57.712187 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:57.712205 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:57.712837 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:57.712910 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:57.712942 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:57.713090 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:57.713353 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 14:48:57.713526 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":128,"threads_started":1}
I20260926 14:48:57.715969 21804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c5017fe204212802cbc2df5a8a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260926 14:48:57.716217 21804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:57.716256 21804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d4c5017fe204212802cbc2df5a8a487, State: Initialized, Role: FOLLOWER
I20260926 14:48:57.716722 21804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c5017fe204212802cbc2df5a8a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260926 14:48:57.716902 21804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:57.716971 21804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:57.717082 21804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:57.717917 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:57.718168 21804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c5017fe204212802cbc2df5a8a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260926 14:48:57.718508 21804 leader_election.cc:302] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d4c5017fe204212802cbc2df5a8a487; no voters:
I20260926 14:48:57.718840 21804 leader_election.cc:288] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:57.718930 21809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:57.719153 21809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [term 1 LEADER]: Becoming Leader. State: Replica: 8d4c5017fe204212802cbc2df5a8a487, State: Running, Role: LEADER
I20260926 14:48:57.719511 21809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c5017fe204212802cbc2df5a8a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } }
I20260926 14:48:57.720166 21804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:57.720952 21811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d4c5017fe204212802cbc2df5a8a487. Latest consensus state: current_term: 1 leader_uuid: "8d4c5017fe204212802cbc2df5a8a487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c5017fe204212802cbc2df5a8a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260926 14:48:57.721110 21811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:57.721450 21823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:57.721459 21810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d4c5017fe204212802cbc2df5a8a487" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d4c5017fe204212802cbc2df5a8a487" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38305 } } }
I20260926 14:48:57.721596 21810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:57.724943 21823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:57.725754 21807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:57.720237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:58081 (local address 127.0.0.1:38305)
0926 14:48:57.720402 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:57.720406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:57.720660 (+ 254us) server_negotiation.cc:407] Connection header received
0926 14:48:57.720787 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:57.720790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:57.720846 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:57.720935 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:57.721853 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:57.722623 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:57.723784 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:48:57.724107 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:48:57.724197 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:48:57.725280 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:57.725284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:57.725287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:57.725300 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:57.725327 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:57.725339 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:57.725386 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:57.725614 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 14:48:57.725652 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 14:48:57.731501 21823 catalog_manager.cc:1347] Generated new cluster ID: 5b12a5eb351c48b09ea89ac67f63940d
I20260926 14:48:57.731577 21823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:57.744881 21823 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:57.745890 21823 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:57.760130 21823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d4c5017fe204212802cbc2df5a8a487: Generated new TSK 0
I20260926 14:48:57.760860 21823 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:57.773669 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 21736
W20260926 14:48:57.780316 21721 connection.cc:535] client connection to 127.0.0.1:38305 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38305): Connection reset by peer (error 104)
2026-09-26T14:48:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:57 dist-test-slave-s7k6 krb5kdc[21835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:57 dist-test-slave-s7k6 krb5kdc[21835](info): set up 2 sockets
Sep 26 14:48:57 dist-test-slave-s7k6 krb5kdc[21835](info): commencing operation
krb5kdc: starting...
W20260926 14:48:58.738901 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:48:58 dist-test-slave-s7k6 krb5kdc[21835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434138, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:48:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:48:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 14:48:58.855224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:48:58.965467 21848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:48:58.965814 21848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:48:58.965878 21848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:48:58.970321 21848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:48:58.970398 21848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:48:58.970430 21848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:48:58.970450 21848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:48:58.970471 21848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:48:58.976140 21848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:48:58.977285 21848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:48:58.978371 21848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:48:58.987001 21856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:58.987012 21854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:48:58.987001 21853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:48:58.987406 21848 server_base.cc:1034] running on GCE node
I20260926 14:48:58.987994 21848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:48:58.989181 21848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:48:58.990339 21848 hybrid_clock.cc:648] HybridClock initialized: now 1790434138990325 us; error 32 us; skew 500 ppm
Sep 26 14:48:58 dist-test-slave-s7k6 krb5kdc[21835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434138, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:48:58.998152 21848 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 14:48:58.999150 21848 webserver.cc:460] Webserver started at http://127.0.0.1:38723/ using document root <none> and password file <none>
I20260926 14:48:58.999707 21848 fs_manager.cc:359] Metadata directory not provided
I20260926 14:48:58.999773 21848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:48:58.999986 21848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:48:59.002130 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "71f2b518de8b4c9388306b186f7f5345"
format_stamp: "Formatted at 2026-09-26 14:48:58 on dist-test-slave-s7k6"
server_key: "9999eee8344b1eeb24b2786c205115df"
server_key_iv: "6a0890bcbd65704330e130662b78ac66"
server_key_version: "encryptionkey@0"
I20260926 14:48:59.002669 21848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "71f2b518de8b4c9388306b186f7f5345"
format_stamp: "Formatted at 2026-09-26 14:48:58 on dist-test-slave-s7k6"
server_key: "9999eee8344b1eeb24b2786c205115df"
server_key_iv: "6a0890bcbd65704330e130662b78ac66"
server_key_version: "encryptionkey@0"
I20260926 14:48:59.006403 21848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:48:59.009025 21863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:48:59.010277 21848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:48:59.010407 21848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "71f2b518de8b4c9388306b186f7f5345"
format_stamp: "Formatted at 2026-09-26 14:48:58 on dist-test-slave-s7k6"
server_key: "9999eee8344b1eeb24b2786c205115df"
server_key_iv: "6a0890bcbd65704330e130662b78ac66"
server_key_version: "encryptionkey@0"
I20260926 14:48:59.010511 21848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:48:59.015245 21848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:48:59.018226 21848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:48:59.018404 21848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:48:59.025537 21848 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44561
I20260926 14:48:59.025557 21915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44561 every 8 connection(s)
I20260926 14:48:59.026484 21848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:48:59.029361 21916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:48:59.031221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 21848
I20260926 14:48:59.031355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:48:59.031653 392 external_mini_cluster.cc:1428] Setting key b3b3c4c21e6134c10e9852460a7b3ff5
I20260926 14:48:59.036180 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345: Bootstrap starting.
I20260926 14:48:59.037056 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:59.033102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:33353 (local address 127.0.0.1:44561)
0926 14:48:59.033596 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:59.033613 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:59.033650 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:48:59.034309 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:59.034328 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:59.034666 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:59.035033 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:59.035860 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:59.035865 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:59.035878 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:59.035888 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:59.035943 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:59.035962 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:59.036090 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:59.036171 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 14:48:59.036335 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":144,"threads_started":1}
I20260926 14:48:59.038666 21916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345: Neither blocks nor log segments found. Creating new log.
I20260926 14:48:59.039562 21916 log.cc:824] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345: Log is configured to *not* fsync() on all Append() calls
I20260926 14:48:59.039891 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:48:59.041543 21916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345: No bootstrap required, opened a new log
I20260926 14:48:59.043380 21919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:48:59.041770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.96:39031 (local address 127.0.0.1:44561)
0926 14:48:59.041867 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:48:59.041871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:48:59.042116 (+ 245us) server_negotiation.cc:407] Connection header received
0926 14:48:59.042230 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:48:59.042233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:48:59.042265 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:48:59.042327 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:48:59.043022 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:48:59.043025 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:48:59.043028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:48:59.043039 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:48:59.043066 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:48:59.043075 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:48:59.043145 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 14:48:59.043228 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 14:48:59.043265 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 14:48:59.045500 21916 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f2b518de8b4c9388306b186f7f5345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260926 14:48:59.045730 21916 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:48:59.045775 21916 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71f2b518de8b4c9388306b186f7f5345, State: Initialized, Role: FOLLOWER
I20260926 14:48:59.046186 21916 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f2b518de8b4c9388306b186f7f5345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260926 14:48:59.046317 21916 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:48:59.046377 21916 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:48:59.046473 21916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:48:59.047420 21916 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f2b518de8b4c9388306b186f7f5345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260926 14:48:59.047767 21916 leader_election.cc:302] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71f2b518de8b4c9388306b186f7f5345; no voters:
I20260926 14:48:59.048154 21916 leader_election.cc:288] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:48:59.048231 21926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:48:59.048432 21926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [term 1 LEADER]: Becoming Leader. State: Replica: 71f2b518de8b4c9388306b186f7f5345, State: Running, Role: LEADER
I20260926 14:48:59.048803 21926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f2b518de8b4c9388306b186f7f5345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } }
I20260926 14:48:59.049396 21916 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:48:59.050442 21928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71f2b518de8b4c9388306b186f7f5345. Latest consensus state: current_term: 1 leader_uuid: "71f2b518de8b4c9388306b186f7f5345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f2b518de8b4c9388306b186f7f5345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } } }
I20260926 14:48:59.050556 21928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:59.050875 21927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71f2b518de8b4c9388306b186f7f5345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71f2b518de8b4c9388306b186f7f5345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44561 } } }
I20260926 14:48:59.050940 21935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:48:59.051025 21927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345 [sys.catalog]: This master's current role is: LEADER
I20260926 14:48:59.054240 21935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:48:59.060019 21935 catalog_manager.cc:1347] Generated new cluster ID: ab3836d4822042e7ad8390889e6f2bfa
I20260926 14:48:59.060168 21935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:48:59.087288 21935 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:48:59.087692 21935 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:48:59.092119 21935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71f2b518de8b4c9388306b186f7f5345: Generated new TSK 0
I20260926 14:48:59.092834 21935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:48:59.098106 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 21848
2026-09-26T14:48:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:48:59 dist-test-slave-s7k6 krb5kdc[21947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:48:59 dist-test-slave-s7k6 krb5kdc[21947](info): set up 2 sockets
Sep 26 14:48:59 dist-test-slave-s7k6 krb5kdc[21947](info): commencing operation
krb5kdc: starting...
W20260926 14:49:00.054797 392 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:49:00 dist-test-slave-s7k6 krb5kdc[21947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:49:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:49:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:49:00.167201 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45167
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:49:00.279277 21960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:00.279551 21960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:00.279592 21960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:00.283676 21960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:00.283738 21960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:00.283757 21960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:00.283772 21960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:00.283787 21960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:00.289052 21960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45167
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:00.290081 21960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:00.291131 21960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:49:00.299659 21968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:00.299654 21966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:00.299708 21960 server_base.cc:1034] running on GCE node
W20260926 14:49:00.299654 21965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:00.300499 21960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:00.301699 21960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:00.302809 21960 hybrid_clock.cc:648] HybridClock initialized: now 1790434140302783 us; error 44 us; skew 500 ppm
Sep 26 14:49:00 dist-test-slave-s7k6 krb5kdc[21947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434140, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:00.305562 21960 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:49:00.306485 21960 webserver.cc:460] Webserver started at http://127.0.98.62:46509/ using document root <none> and password file <none>
I20260926 14:49:00.307071 21960 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:00.307139 21960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:00.307356 21960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:00.309374 21960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "8b861acdb00d40da82d87cb3d075eef4"
format_stamp: "Formatted at 2026-09-26 14:49:00 on dist-test-slave-s7k6"
server_key: "daed4fc6389730a410f098facdb53f47"
server_key_iv: "d5cb115d292f8880b2b5cac2b21ae51b"
server_key_version: "encryptionkey@0"
I20260926 14:49:00.309911 21960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b861acdb00d40da82d87cb3d075eef4"
format_stamp: "Formatted at 2026-09-26 14:49:00 on dist-test-slave-s7k6"
server_key: "daed4fc6389730a410f098facdb53f47"
server_key_iv: "d5cb115d292f8880b2b5cac2b21ae51b"
server_key_version: "encryptionkey@0"
I20260926 14:49:00.313671 21960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 14:49:00.316272 21975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:00.317745 21960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 14:49:00.317870 21960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "8b861acdb00d40da82d87cb3d075eef4"
format_stamp: "Formatted at 2026-09-26 14:49:00 on dist-test-slave-s7k6"
server_key: "daed4fc6389730a410f098facdb53f47"
server_key_iv: "d5cb115d292f8880b2b5cac2b21ae51b"
server_key_version: "encryptionkey@0"
I20260926 14:49:00.317978 21960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:00.343796 21960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:00.347252 21960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:00.347455 21960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:00.354460 21960 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45167
I20260926 14:49:00.354478 22027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45167 every 8 connection(s)
I20260926 14:49:00.355494 21960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:49:00.358289 22028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:00.363821 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 21960
I20260926 14:49:00.363960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:49:00.364161 22028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4: Bootstrap starting.
I20260926 14:49:00.364230 392 external_mini_cluster.cc:1428] Setting key f0c765ec12bd1a8e3adab2d0e79f156d
I20260926 14:49:00.366693 22028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:00.367471 22028 log.cc:824] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:00.369482 22028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4: No bootstrap required, opened a new log
Sep 26 14:49:00 dist-test-slave-s7k6 krb5kdc[21947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:49:00.373894 22028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b861acdb00d40da82d87cb3d075eef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } }
I20260926 14:49:00.374159 22028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:49:00.374217 22028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b861acdb00d40da82d87cb3d075eef4, State: Initialized, Role: FOLLOWER
I20260926 14:49:00.374743 22028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b861acdb00d40da82d87cb3d075eef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } }
I20260926 14:49:00.374943 22028 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:49:00.375008 22028 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:49:00.375161 22028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:49:00.376178 22028 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b861acdb00d40da82d87cb3d075eef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } }
I20260926 14:49:00.376610 22028 leader_election.cc:302] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b861acdb00d40da82d87cb3d075eef4; no voters:
I20260926 14:49:00.377008 22028 leader_election.cc:288] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:49:00.377102 22033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:49:00.377311 22033 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [term 1 LEADER]: Becoming Leader. State: Replica: 8b861acdb00d40da82d87cb3d075eef4, State: Running, Role: LEADER
I20260926 14:49:00.377727 22033 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b861acdb00d40da82d87cb3d075eef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } }
I20260926 14:49:00.378473 22028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:00.379201 22033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b861acdb00d40da82d87cb3d075eef4. Latest consensus state: current_term: 1 leader_uuid: "8b861acdb00d40da82d87cb3d075eef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b861acdb00d40da82d87cb3d075eef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } } }
I20260926 14:49:00.379320 22033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:00.379565 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:00.365602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51153 (local address 127.0.98.62:45167)
0926 14:49:00.366011 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:00.366027 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:00.366064 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:49:00.366729 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:00.366749 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:00.367176 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:00.367488 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:00.368323 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:00.369333 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:00.370480 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:00.371001 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:00.371120 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:00.373998 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:00.374002 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:00.374016 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:00.374046 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:00.375879 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:00.376492 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:00.376497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:00.376502 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:00.376630 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:00.377020 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:00.377023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:00.377025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:00.377623 (+ 598us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:00.377849 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:00.378205 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 14:49:00.378480 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":114,"threads_started":1}
I20260926 14:49:00.379734 22034 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b861acdb00d40da82d87cb3d075eef4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b861acdb00d40da82d87cb3d075eef4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45167 } } }
I20260926 14:49:00.379914 22034 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:00.380820 22041 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:49:00.383170 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:49:00.384020 22041 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:49:00.389986 22041 catalog_manager.cc:1347] Generated new cluster ID: ca8d539771e84ed9b0465f35e5607383
I20260926 14:49:00.390053 22041 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:49:00.394671 22031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:00.385469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51513 (local address 127.0.98.62:45167)
0926 14:49:00.385586 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:00.385591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:00.385894 (+ 303us) server_negotiation.cc:407] Connection header received
0926 14:49:00.386049 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:00.386052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:00.386102 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:00.386179 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:00.387089 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:00.387804 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:00.388716 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:00.388976 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:00.389065 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:00.390821 (+ 1756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:00.390825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:00.390828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:00.390858 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:00.392568 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:00.393106 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:00.393110 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:00.393112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:00.393203 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:00.393620 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:00.393624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:00.393626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:00.393844 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:00.393988 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:00.394416 (+ 428us) server_negotiation.cc:299] Negotiation successful
0926 14:49:00.394516 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 14:49:00.419723 22041 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:49:00.420797 22041 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:49:00.428634 22041 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b861acdb00d40da82d87cb3d075eef4: Generated new TSK 0
I20260926 14:49:00.429420 22041 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:49:00.434614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 21960
W20260926 14:49:00.439358 21945 connection.cc:535] client connection to 127.0.98.62:45167 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:45167): Connection reset by peer (error 104)
2026-09-26T14:49:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1336 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:49:00 dist-test-slave-s7k6 krb5kdc[22059](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:49:00 dist-test-slave-s7k6 krb5kdc[22059](info): set up 2 sockets
Sep 26 14:49:00 dist-test-slave-s7k6 krb5kdc[22059](info): commencing operation
krb5kdc: starting...
W20260926 14:49:01.378860 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:49:01 dist-test-slave-s7k6 krb5kdc[22059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:49:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:49:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:49:01.498492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35227
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35227
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:49:01.610338 22072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:01.610618 22072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:01.610661 22072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:01.614811 22072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:01.614892 22072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:01.614912 22072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:01.614926 22072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:01.614940 22072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:01.621474 22072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35227
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:01.622663 22072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:01.623791 22072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:49:01.632584 22078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:01.632582 22077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:01.632584 22080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:01.633093 22072 server_base.cc:1034] running on GCE node
I20260926 14:49:01.633665 22072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:01.634908 22072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:01.636106 22072 hybrid_clock.cc:648] HybridClock initialized: now 1790434141636087 us; error 35 us; skew 500 ppm
Sep 26 14:49:01 dist-test-slave-s7k6 krb5kdc[22059](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434141, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:01.638921 22072 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:49:01.639868 22072 webserver.cc:460] Webserver started at http://127.0.98.62:46863/ using document root <none> and password file <none>
I20260926 14:49:01.640436 22072 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:01.640506 22072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:01.640728 22072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:01.642830 22072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "02365a3bebbc4f7eacc4435d082ccfc5"
format_stamp: "Formatted at 2026-09-26 14:49:01 on dist-test-slave-s7k6"
server_key: "fa0d8cc5aa075878e1e0eb730b0c0b22"
server_key_iv: "966ebee8e7a22b8abbcb7ac94684d767"
server_key_version: "encryptionkey@0"
I20260926 14:49:01.643361 22072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "02365a3bebbc4f7eacc4435d082ccfc5"
format_stamp: "Formatted at 2026-09-26 14:49:01 on dist-test-slave-s7k6"
server_key: "fa0d8cc5aa075878e1e0eb730b0c0b22"
server_key_iv: "966ebee8e7a22b8abbcb7ac94684d767"
server_key_version: "encryptionkey@0"
I20260926 14:49:01.647174 22072 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 14:49:01.649657 22087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:01.651120 22072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:49:01.651276 22072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "02365a3bebbc4f7eacc4435d082ccfc5"
format_stamp: "Formatted at 2026-09-26 14:49:01 on dist-test-slave-s7k6"
server_key: "fa0d8cc5aa075878e1e0eb730b0c0b22"
server_key_iv: "966ebee8e7a22b8abbcb7ac94684d767"
server_key_version: "encryptionkey@0"
I20260926 14:49:01.651391 22072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:01.680403 22072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:01.683503 22072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:01.683696 22072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:01.690577 22072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35227
I20260926 14:49:01.690591 22139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35227 every 8 connection(s)
I20260926 14:49:01.691576 22072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:49:01.694487 22140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:01.694980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 22072
I20260926 14:49:01.695155 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:49:01.695438 392 external_mini_cluster.cc:1428] Setting key d027a6ef802d7252cbcac15921262108
I20260926 14:49:01.701897 22140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5: Bootstrap starting.
I20260926 14:49:01.704579 22140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:01.704980 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:01.696800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47093 (local address 127.0.98.62:35227)
0926 14:49:01.697261 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:01.697280 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:01.697317 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:49:01.697998 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:01.698017 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:01.698390 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:01.698699 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:01.699081 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:01.700286 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:01.701508 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:01.702003 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:01.702179 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:01.703190 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:01.703196 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:01.703209 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:49:01.703602 (+ 393us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:01.703662 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:49:01.703694 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:01.703798 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:01.703988 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 14:49:01.704179 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":312,"thread_start_us":149,"threads_started":1}
I20260926 14:49:01.705331 22140 log.cc:824] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:01.707557 22140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5: No bootstrap required, opened a new log
I20260926 14:49:01.708305 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:49:01.712044 22140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02365a3bebbc4f7eacc4435d082ccfc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260926 14:49:01.712352 22140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:49:01.712422 22140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02365a3bebbc4f7eacc4435d082ccfc5, State: Initialized, Role: FOLLOWER
I20260926 14:49:01.712915 22140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02365a3bebbc4f7eacc4435d082ccfc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260926 14:49:01.713078 22140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:49:01.713151 22140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:49:01.713251 22140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:49:01.714411 22140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02365a3bebbc4f7eacc4435d082ccfc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260926 14:49:01.714802 22140 leader_election.cc:302] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02365a3bebbc4f7eacc4435d082ccfc5; no voters:
I20260926 14:49:01.714969 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:01.710552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33173 (local address 127.0.98.62:35227)
0926 14:49:01.710668 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:01.710673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:01.710817 (+ 144us) server_negotiation.cc:407] Connection header received
0926 14:49:01.710974 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:01.710977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:01.711085 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:01.711161 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:01.711458 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:01.711981 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:01.713120 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:01.713462 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:01.713573 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:01.714408 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:01.714411 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:01.714413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:49:01.714424 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:01.714449 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:49:01.714459 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:01.714573 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:01.714719 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 14:49:01.714794 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:49:01.715173 22140 leader_election.cc:288] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:49:01.715260 22150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:49:01.715452 22150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [term 1 LEADER]: Becoming Leader. State: Replica: 02365a3bebbc4f7eacc4435d082ccfc5, State: Running, Role: LEADER
I20260926 14:49:01.715771 22150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02365a3bebbc4f7eacc4435d082ccfc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } }
I20260926 14:49:01.716460 22140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:01.716996 22150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02365a3bebbc4f7eacc4435d082ccfc5. Latest consensus state: current_term: 1 leader_uuid: "02365a3bebbc4f7eacc4435d082ccfc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02365a3bebbc4f7eacc4435d082ccfc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } } }
I20260926 14:49:01.717118 22150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:01.717512 22151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02365a3bebbc4f7eacc4435d082ccfc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02365a3bebbc4f7eacc4435d082ccfc5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35227 } } }
I20260926 14:49:01.717643 22151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:01.718740 22154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:49:01.721501 22154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:49:01.727238 22154 catalog_manager.cc:1347] Generated new cluster ID: 68f9ca949cbf43bb9f1a324f8cef1d85
I20260926 14:49:01.727317 22154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:49:01.759502 22154 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:49:01.760946 22154 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:49:01.769457 22154 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02365a3bebbc4f7eacc4435d082ccfc5: Generated new TSK 0
I20260926 14:49:01.770337 22154 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:49:01.774430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 22072
W20260926 14:49:01.779452 22057 connection.cc:535] client connection to 127.0.98.62:35227 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35227): Connection reset by peer (error 104)
2026-09-26T14:49:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:49:01 dist-test-slave-s7k6 krb5kdc[22171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:49:01 dist-test-slave-s7k6 krb5kdc[22171](info): set up 2 sockets
Sep 26 14:49:01 dist-test-slave-s7k6 krb5kdc[22171](info): commencing operation
krb5kdc: starting...
W20260926 14:49:02.727129 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:49:02 dist-test-slave-s7k6 krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:49:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:49:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:49:02.849326 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36485
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:49:02.962935 22184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:02.963234 22184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:02.963279 22184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:02.967392 22184 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:02.967463 22184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:02.967483 22184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:02.967499 22184 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:02.967514 22184 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:02.973081 22184 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36485
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:02.974196 22184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:02.975309 22184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:49:02.984350 22192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:02.984478 22184 server_base.cc:1034] running on GCE node
W20260926 14:49:02.984375 22189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:02.984355 22190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:02.985296 22184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:02.986531 22184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:02.987703 22184 hybrid_clock.cc:648] HybridClock initialized: now 1790434142987687 us; error 30 us; skew 500 ppm
Sep 26 14:49:02 dist-test-slave-s7k6 krb5kdc[22171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434142, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:02.990510 22184 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:49:02.991479 22184 webserver.cc:460] Webserver started at http://127.0.98.62:46659/ using document root <none> and password file <none>
I20260926 14:49:02.992044 22184 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:02.992113 22184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:02.992303 22184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:02.994462 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "5c6d7faa1f134fcbbd14fc5c23e510a2"
format_stamp: "Formatted at 2026-09-26 14:49:02 on dist-test-slave-s7k6"
server_key: "28ec97d12bf3830888dd6e424dbf3959"
server_key_iv: "32215bfe53a7a75790705cffbf968da6"
server_key_version: "encryptionkey@0"
I20260926 14:49:02.995056 22184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "5c6d7faa1f134fcbbd14fc5c23e510a2"
format_stamp: "Formatted at 2026-09-26 14:49:02 on dist-test-slave-s7k6"
server_key: "28ec97d12bf3830888dd6e424dbf3959"
server_key_iv: "32215bfe53a7a75790705cffbf968da6"
server_key_version: "encryptionkey@0"
I20260926 14:49:02.998804 22184 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 14:49:03.001323 22199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:03.002709 22184 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 14:49:03.002849 22184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "5c6d7faa1f134fcbbd14fc5c23e510a2"
format_stamp: "Formatted at 2026-09-26 14:49:02 on dist-test-slave-s7k6"
server_key: "28ec97d12bf3830888dd6e424dbf3959"
server_key_iv: "32215bfe53a7a75790705cffbf968da6"
server_key_version: "encryptionkey@0"
I20260926 14:49:03.002960 22184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:03.007570 22184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:03.011060 22184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:03.011242 22184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:03.018256 22184 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36485
I20260926 14:49:03.018288 22251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36485 every 8 connection(s)
I20260926 14:49:03.019270 22184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:49:03.022298 22252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:03.025499 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 22184
I20260926 14:49:03.025655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:49:03.025967 392 external_mini_cluster.cc:1428] Setting key 02c6bdfb01d9a922a2f7446867951373
I20260926 14:49:03.028743 22252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2: Bootstrap starting.
I20260926 14:49:03.031226 22252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:03.031669 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:03.027480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48965 (local address 127.0.98.62:36485)
0926 14:49:03.027947 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:03.027963 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:03.028000 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:49:03.028712 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:03.028731 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:03.029086 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:03.029423 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:03.030244 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:03.030248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:03.030261 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:49:03.030271 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:03.030327 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:49:03.030347 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:03.030477 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:03.030548 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 14:49:03.030710 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":146,"threads_started":1}
I20260926 14:49:03.032042 22252 log.cc:824] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:03.034052 22252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2: No bootstrap required, opened a new log
I20260926 14:49:03.034685 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:49:03.037519 22252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6d7faa1f134fcbbd14fc5c23e510a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } }
I20260926 14:49:03.037794 22252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:49:03.037885 22252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c6d7faa1f134fcbbd14fc5c23e510a2, State: Initialized, Role: FOLLOWER
I20260926 14:49:03.038378 22252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6d7faa1f134fcbbd14fc5c23e510a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } }
I20260926 14:49:03.038561 22252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:49:03.038651 22252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:49:03.038700 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:03.037077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58303 (local address 127.0.98.62:36485)
0926 14:49:03.037187 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:03.037190 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:03.037202 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:49:03.037234 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:03.037236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:03.037262 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:03.037324 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:03.038212 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:03.038215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:03.038217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:49:03.038228 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:03.038256 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:49:03.038267 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:03.038345 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:03.038477 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 14:49:03.038527 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 14:49:03.038812 22252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:49:03.039786 22252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6d7faa1f134fcbbd14fc5c23e510a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } }
I20260926 14:49:03.040123 22252 leader_election.cc:302] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c6d7faa1f134fcbbd14fc5c23e510a2; no voters:
I20260926 14:49:03.040414 22252 leader_election.cc:288] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:49:03.040592 22262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:49:03.040794 22262 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [term 1 LEADER]: Becoming Leader. State: Replica: 5c6d7faa1f134fcbbd14fc5c23e510a2, State: Running, Role: LEADER
I20260926 14:49:03.041096 22262 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6d7faa1f134fcbbd14fc5c23e510a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } }
I20260926 14:49:03.041628 22252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:03.042833 22264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c6d7faa1f134fcbbd14fc5c23e510a2. Latest consensus state: current_term: 1 leader_uuid: "5c6d7faa1f134fcbbd14fc5c23e510a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6d7faa1f134fcbbd14fc5c23e510a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } } }
I20260926 14:49:03.042833 22263 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c6d7faa1f134fcbbd14fc5c23e510a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c6d7faa1f134fcbbd14fc5c23e510a2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36485 } } }
I20260926 14:49:03.043021 22264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:03.043049 22263 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:03.043418 22272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:49:03.046907 22272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:49:03.052816 22272 catalog_manager.cc:1347] Generated new cluster ID: 4f40168d2ce94988bc79461494001ac9
I20260926 14:49:03.052889 22272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:49:03.063758 22272 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:49:03.064106 22272 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:49:03.071650 22272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c6d7faa1f134fcbbd14fc5c23e510a2: Generated new TSK 0
I20260926 14:49:03.072381 22272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:49:03.084836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 22184
2026-09-26T14:49:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1309 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:49:03 dist-test-slave-s7k6 krb5kdc[22283](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:49:03 dist-test-slave-s7k6 krb5kdc[22283](info): set up 2 sockets
Sep 26 14:49:03 dist-test-slave-s7k6 krb5kdc[22283](info): commencing operation
krb5kdc: starting...
W20260926 14:49:04.034960 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:49:04 dist-test-slave-s7k6 krb5kdc[22283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:49:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:49:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:49:04.162412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42541
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:49:04.277271 22296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:04.277591 22296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:04.277657 22296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:04.281845 22296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:04.281917 22296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:04.281941 22296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:04.281957 22296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:04.281970 22296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:04.287624 22296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:04.288781 22296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:04.289855 22296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:49:04.298944 22302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:04.298969 22301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:04.299058 22296 server_base.cc:1034] running on GCE node
W20260926 14:49:04.298921 22304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:04.299786 22296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:04.300981 22296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:04.302186 22296 hybrid_clock.cc:648] HybridClock initialized: now 1790434144302150 us; error 50 us; skew 500 ppm
Sep 26 14:49:04 dist-test-slave-s7k6 krb5kdc[22283](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:04.312918 22296 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:49:04.313891 22296 webserver.cc:460] Webserver started at http://127.0.98.62:41377/ using document root <none> and password file <none>
I20260926 14:49:04.314570 22296 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:04.314651 22296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:04.314891 22296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:04.317123 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "747696d4c44646afa187f6e16e20fc13"
format_stamp: "Formatted at 2026-09-26 14:49:04 on dist-test-slave-s7k6"
server_key: "220435db7f32f1d2d2b13b56ece49d7b"
server_key_iv: "a6664beb9e6c2b01c2c457f6b22aef0c"
server_key_version: "encryptionkey@0"
I20260926 14:49:04.317739 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "747696d4c44646afa187f6e16e20fc13"
format_stamp: "Formatted at 2026-09-26 14:49:04 on dist-test-slave-s7k6"
server_key: "220435db7f32f1d2d2b13b56ece49d7b"
server_key_iv: "a6664beb9e6c2b01c2c457f6b22aef0c"
server_key_version: "encryptionkey@0"
I20260926 14:49:04.321795 22296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 14:49:04.324630 22311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:04.326004 22296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260926 14:49:04.326133 22296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "747696d4c44646afa187f6e16e20fc13"
format_stamp: "Formatted at 2026-09-26 14:49:04 on dist-test-slave-s7k6"
server_key: "220435db7f32f1d2d2b13b56ece49d7b"
server_key_iv: "a6664beb9e6c2b01c2c457f6b22aef0c"
server_key_version: "encryptionkey@0"
I20260926 14:49:04.326251 22296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:04.345875 22296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:04.349208 22296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:04.349409 22296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:04.356775 22296 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42541
I20260926 14:49:04.356812 22363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42541 every 8 connection(s)
I20260926 14:49:04.357810 22296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:49:04.359072 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 22296
I20260926 14:49:04.359223 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:49:04.359530 392 external_mini_cluster.cc:1428] Setting key 082e1ff15518dbf8f89b117cc6ceb751
I20260926 14:49:04.361549 22364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:04.367916 22364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13: Bootstrap starting.
Sep 26 14:49:04 dist-test-slave-s7k6 krb5kdc[22283](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:49:04.370421 22364 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:04.371268 22364 log.cc:824] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:04.373217 22364 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13: No bootstrap required, opened a new log
I20260926 14:49:04.374269 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:04.361067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56347 (local address 127.0.98.62:42541)
0926 14:49:04.361600 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:04.361621 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:04.361672 (+ 51us) server_negotiation.cc:407] Connection header received
0926 14:49:04.362619 (+ 947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:04.362649 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:04.363187 (+ 538us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:04.363560 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:04.364410 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:04.365452 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:04.366353 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:04.366932 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:04.367082 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:04.369589 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:04.369595 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:04.369608 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:04.369640 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:04.371410 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:04.371857 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:04.371862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:04.371868 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:04.371986 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:04.372260 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:04.372263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:04.372264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:04.372676 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:04.372858 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:04.373157 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 14:49:04.373476 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":154,"threads_started":1}
I20260926 14:49:04.376518 22364 raft_consensus.cc:348] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747696d4c44646afa187f6e16e20fc13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42541 } }
I20260926 14:49:04.376824 22364 raft_consensus.cc:374] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:49:04.376889 22364 raft_consensus.cc:729] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 747696d4c44646afa187f6e16e20fc13, State: Initialized, Role: FOLLOWER
I20260926 14:49:04.377382 22364 consensus_queue.cc:260] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747696d4c44646afa187f6e16e20fc13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42541 } }
I20260926 14:49:04.377519 22364 raft_consensus.cc:388] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:49:04.377580 22364 raft_consensus.cc:482] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:49:04.377682 22364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:49:04.377759 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:49:04.378690 22364 raft_consensus.cc:504] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747696d4c44646afa187f6e16e20fc13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42541 } }
I20260926 14:49:04.379109 22364 leader_election.cc:302] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 747696d4c44646afa187f6e16e20fc13; no voters:
I20260926 14:49:04.379449 22364 leader_election.cc:288] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:49:04.379573 22373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:49:04.379829 22373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [term 1 LEADER]: Becoming Leader. State: Replica: 747696d4c44646afa187f6e16e20fc13, State: Running, Role: LEADER
I20260926 14:49:04.380275 22373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747696d4c44646afa187f6e16e20fc13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42541 } }
I20260926 14:49:04.380946 22364 sys_catalog.cc:564] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:04.381654 22373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 747696d4c44646afa187f6e16e20fc13. Latest consensus state: current_term: 1 leader_uuid: "747696d4c44646afa187f6e16e20fc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747696d4c44646afa187f6e16e20fc13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42541 } } }
I20260926 14:49:04.381848 22373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:04.382076 22374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "747696d4c44646afa187f6e16e20fc13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "747696d4c44646afa187f6e16e20fc13" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42541 } } }
I20260926 14:49:04.382164 22374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:04.382285 22383 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:49:04.385775 22383 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:49:04.389362 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:04.379841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54127 (local address 127.0.98.62:42541)
0926 14:49:04.379946 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:04.379951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:04.380364 (+ 413us) server_negotiation.cc:407] Connection header received
0926 14:49:04.380524 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:04.380527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:04.380581 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:04.380668 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:04.381489 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:04.382311 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:04.383492 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:04.383822 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:04.383937 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:04.385563 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:04.385567 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:04.385570 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:04.385598 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:04.387371 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:04.388000 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:04.388005 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:04.388007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:04.388110 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:04.388443 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:04.388446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:04.388448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:04.388681 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:04.388826 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:04.389067 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 14:49:04.389199 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 14:49:04.392099 22383 catalog_manager.cc:1347] Generated new cluster ID: f09c0219007e400da60277a6acd491b6
I20260926 14:49:04.392158 22383 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:49:04.405637 22383 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:49:04.406606 22383 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:49:04.414978 22383 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 747696d4c44646afa187f6e16e20fc13: Generated new TSK 0
I20260926 14:49:04.415716 22383 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:49:04.423919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 22296
2026-09-26T14:49:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:49:04 dist-test-slave-s7k6 krb5kdc[22395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:49:04 dist-test-slave-s7k6 krb5kdc[22395](info): set up 2 sockets
Sep 26 14:49:04 dist-test-slave-s7k6 krb5kdc[22395](info): commencing operation
krb5kdc: starting...
W20260926 14:49:05.386861 392 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:49:05 dist-test-slave-s7k6 krb5kdc[22395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:49:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:49:05.505774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:49:05.618907 22408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:05.619236 22408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:05.619285 22408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:05.623428 22408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:05.623499 22408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:05.623518 22408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:05.623533 22408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:05.623548 22408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:05.629308 22408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:05.630484 22408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:05.631987 22408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:49:05.641484 22416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:05.641492 22413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:05.641549 22408 server_base.cc:1034] running on GCE node
W20260926 14:49:05.641491 22414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:05.642282 22408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:05.643466 22408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:05.644625 22408 hybrid_clock.cc:648] HybridClock initialized: now 1790434145644603 us; error 38 us; skew 500 ppm
Sep 26 14:49:05 dist-test-slave-s7k6 krb5kdc[22395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434145, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:05.647528 22408 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:49:05.648474 22408 webserver.cc:460] Webserver started at http://127.0.98.62:44859/ using document root <none> and password file <none>
I20260926 14:49:05.649026 22408 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:05.649093 22408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:05.649305 22408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:05.651481 22408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "7a0d07134cd34a7a9a02a72a6048be53"
format_stamp: "Formatted at 2026-09-26 14:49:05 on dist-test-slave-s7k6"
server_key: "95c868c029c32854b9bed8be06c5ee0c"
server_key_iv: "ee36abca979a40018288c34b058ed321"
server_key_version: "encryptionkey@0"
I20260926 14:49:05.652050 22408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a0d07134cd34a7a9a02a72a6048be53"
format_stamp: "Formatted at 2026-09-26 14:49:05 on dist-test-slave-s7k6"
server_key: "95c868c029c32854b9bed8be06c5ee0c"
server_key_iv: "ee36abca979a40018288c34b058ed321"
server_key_version: "encryptionkey@0"
I20260926 14:49:05.656002 22408 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 14:49:05.658823 22423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:05.660303 22408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 14:49:05.660444 22408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "7a0d07134cd34a7a9a02a72a6048be53"
format_stamp: "Formatted at 2026-09-26 14:49:05 on dist-test-slave-s7k6"
server_key: "95c868c029c32854b9bed8be06c5ee0c"
server_key_iv: "ee36abca979a40018288c34b058ed321"
server_key_version: "encryptionkey@0"
I20260926 14:49:05.660570 22408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:05.686923 22408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:05.690078 22408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:05.690266 22408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:05.697563 22408 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38065
I20260926 14:49:05.697583 22475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38065 every 8 connection(s)
I20260926 14:49:05.698592 22408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:49:05.701633 22476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:05.702257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 22408
I20260926 14:49:05.702389 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:49:05.702734 392 external_mini_cluster.cc:1428] Setting key bfe242ea03e9027e9394f2942cefc426
I20260926 14:49:05.708179 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53: Bootstrap starting.
I20260926 14:49:05.710608 22476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:05.711470 22476 log.cc:824] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:05.712045 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:05.704322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36017 (local address 127.0.98.62:38065)
0926 14:49:05.704788 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:05.704804 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:05.704840 (+ 36us) server_negotiation.cc:407] Connection header received
0926 14:49:05.705537 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:05.705557 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:05.705927 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:05.706254 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:05.706668 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:05.707717 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:05.708641 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:05.709121 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:05.709280 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:05.710142 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:05.710148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:05.710161 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:49:05.710582 (+ 421us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:05.710640 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:49:05.710665 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:05.710797 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:05.710931 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 14:49:05.711167 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":135,"threads_started":1}
I20260926 14:49:05.713492 22476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53: No bootstrap required, opened a new log
I20260926 14:49:05.714893 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:49:05.717415 22476 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0d07134cd34a7a9a02a72a6048be53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } }
I20260926 14:49:05.717705 22476 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:49:05.717770 22476 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a0d07134cd34a7a9a02a72a6048be53, State: Initialized, Role: FOLLOWER
I20260926 14:49:05.718290 22476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0d07134cd34a7a9a02a72a6048be53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } }
I20260926 14:49:05.718443 22476 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:49:05.718504 22476 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:49:05.718608 22476 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:49:05.719615 22476 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0d07134cd34a7a9a02a72a6048be53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } }
I20260926 14:49:05.720000 22476 leader_election.cc:302] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a0d07134cd34a7a9a02a72a6048be53; no voters:
I20260926 14:49:05.720299 22476 leader_election.cc:288] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:49:05.720628 22487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:49:05.720880 22487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [term 1 LEADER]: Becoming Leader. State: Replica: 7a0d07134cd34a7a9a02a72a6048be53, State: Running, Role: LEADER
I20260926 14:49:05.721345 22479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:05.716885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50793 (local address 127.0.98.62:38065)
0926 14:49:05.716988 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:05.716991 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:05.717273 (+ 282us) server_negotiation.cc:407] Connection header received
0926 14:49:05.717371 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:05.717374 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:05.717423 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:05.717505 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:05.717720 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:05.718482 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:05.719454 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:05.719796 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:05.719903 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:05.720876 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:05.720880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:05.720882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:49:05.720897 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:05.720927 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:49:05.720942 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:05.721014 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:05.721164 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 14:49:05.721221 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:49:05.721319 22487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0d07134cd34a7a9a02a72a6048be53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } }
I20260926 14:49:05.721611 22476 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:05.723272 22488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a0d07134cd34a7a9a02a72a6048be53. Latest consensus state: current_term: 1 leader_uuid: "7a0d07134cd34a7a9a02a72a6048be53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0d07134cd34a7a9a02a72a6048be53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } } }
I20260926 14:49:05.723286 22486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a0d07134cd34a7a9a02a72a6048be53" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a0d07134cd34a7a9a02a72a6048be53" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38065 } } }
I20260926 14:49:05.723428 22488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:05.723431 22486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53 [sys.catalog]: This master's current role is: LEADER
W20260926 14:49:05.726408 22501 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:49:05.726475 22501 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:49:05.726553 22500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:49:05.727304 22500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:49:05.733085 22500 catalog_manager.cc:1347] Generated new cluster ID: 27b83995e4cf4cf4962489d4aafe6737
I20260926 14:49:05.733161 22500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:49:05.744810 22500 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:49:05.745855 22500 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:49:05.754103 22500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a0d07134cd34a7a9a02a72a6048be53: Generated new TSK 0
I20260926 14:49:05.754949 22500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:49:05.759109 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 22408
2026-09-26T14:49:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1335 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:49:05 dist-test-slave-s7k6 krb5kdc[22507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:49:05 dist-test-slave-s7k6 krb5kdc[22507](info): set up 2 sockets
Sep 26 14:49:05 dist-test-slave-s7k6 krb5kdc[22507](info): commencing operation
krb5kdc: starting...
W20260926 14:49:06.734988 392 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:49:06 dist-test-slave-s7k6 krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:49:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:49:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:49:06.858526 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34171
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34171
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:49:06.977399 22520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:06.977752 22520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:06.977816 22520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:06.982106 22520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:06.982199 22520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:06.982226 22520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:06.982249 22520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:06.982270 22520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:06.988198 22520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34171
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:06.989428 22520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:06.990553 22520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:49:06.999440 22526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:06.999416 22525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:06.999428 22528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:06.999789 22520 server_base.cc:1034] running on GCE node
I20260926 14:49:07.000391 22520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:07.001627 22520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:07.002815 22520 hybrid_clock.cc:648] HybridClock initialized: now 1790434147002793 us; error 33 us; skew 500 ppm
Sep 26 14:49:06 dist-test-slave-s7k6 krb5kdc[22507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434146, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:07.005685 22520 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:49:07.006620 22520 webserver.cc:460] Webserver started at http://127.0.98.62:40673/ using document root <none> and password file <none>
I20260926 14:49:07.007257 22520 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:07.007330 22520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:07.007548 22520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:07.009706 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "7d8c74a3ffbf46ffb8c20b451c841e0a"
format_stamp: "Formatted at 2026-09-26 14:49:07 on dist-test-slave-s7k6"
server_key: "13ed7c8b588e718fa2bca11748b3b134"
server_key_iv: "0388bbcfacce54090f34ef8d713c246b"
server_key_version: "encryptionkey@0"
I20260926 14:49:07.010273 22520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "7d8c74a3ffbf46ffb8c20b451c841e0a"
format_stamp: "Formatted at 2026-09-26 14:49:07 on dist-test-slave-s7k6"
server_key: "13ed7c8b588e718fa2bca11748b3b134"
server_key_iv: "0388bbcfacce54090f34ef8d713c246b"
server_key_version: "encryptionkey@0"
I20260926 14:49:07.014325 22520 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 14:49:07.017472 22535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:07.018962 22520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:49:07.019099 22520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "7d8c74a3ffbf46ffb8c20b451c841e0a"
format_stamp: "Formatted at 2026-09-26 14:49:07 on dist-test-slave-s7k6"
server_key: "13ed7c8b588e718fa2bca11748b3b134"
server_key_iv: "0388bbcfacce54090f34ef8d713c246b"
server_key_version: "encryptionkey@0"
I20260926 14:49:07.019222 22520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:07.024302 22520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:07.028209 22520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:07.028404 22520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:07.035750 22520 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34171
I20260926 14:49:07.035804 22587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34171 every 8 connection(s)
I20260926 14:49:07.036759 22520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:49:07.039860 22588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:07.045406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 22520
I20260926 14:49:07.045565 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:49:07.045859 392 external_mini_cluster.cc:1428] Setting key 39c756a172a45ba588968b3d62999b1e
I20260926 14:49:07.045974 22588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a: Bootstrap starting.
I20260926 14:49:07.048707 22588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:07.049590 22588 log.cc:824] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:07.051499 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:07.047295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35917 (local address 127.0.98.62:34171)
0926 14:49:07.047770 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:07.047786 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:07.047823 (+ 37us) server_negotiation.cc:407] Connection header received
0926 14:49:07.048552 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:07.048579 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:07.048937 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:07.049264 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:07.050152 (+ 888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:07.050158 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:07.050175 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:49:07.050188 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:07.050254 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:49:07.050281 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:07.050395 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:07.050520 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 14:49:07.050695 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":142,"threads_started":1}
I20260926 14:49:07.051992 22588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a: No bootstrap required, opened a new log
I20260926 14:49:07.054728 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:49:07.056763 22588 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8c74a3ffbf46ffb8c20b451c841e0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260926 14:49:07.057080 22588 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 14:49:07.057144 22588 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d8c74a3ffbf46ffb8c20b451c841e0a, State: Initialized, Role: FOLLOWER
I20260926 14:49:07.057706 22588 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8c74a3ffbf46ffb8c20b451c841e0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260926 14:49:07.057883 22588 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 14:49:07.057956 22588 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 14:49:07.058070 22588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:49:07.059024 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:07.057258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58855 (local address 127.0.98.62:34171)
0926 14:49:07.057477 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:07.057483 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:07.057624 (+ 141us) server_negotiation.cc:407] Connection header received
0926 14:49:07.057759 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:07.057761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:07.057795 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:07.057879 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:07.058641 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:07.058645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:07.058647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 14:49:07.058657 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:07.058682 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 14:49:07.058693 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:07.058787 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:07.058870 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 14:49:07.058906 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:49:07.059389 22588 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8c74a3ffbf46ffb8c20b451c841e0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260926 14:49:07.059798 22588 leader_election.cc:302] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d8c74a3ffbf46ffb8c20b451c841e0a; no voters:
I20260926 14:49:07.060209 22588 leader_election.cc:288] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 14:49:07.060278 22598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:49:07.060506 22598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [term 1 LEADER]: Becoming Leader. State: Replica: 7d8c74a3ffbf46ffb8c20b451c841e0a, State: Running, Role: LEADER
I20260926 14:49:07.060820 22598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8c74a3ffbf46ffb8c20b451c841e0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } }
I20260926 14:49:07.061480 22588 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:07.062657 22600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d8c74a3ffbf46ffb8c20b451c841e0a. Latest consensus state: current_term: 1 leader_uuid: "7d8c74a3ffbf46ffb8c20b451c841e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8c74a3ffbf46ffb8c20b451c841e0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } }
I20260926 14:49:07.062690 22599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d8c74a3ffbf46ffb8c20b451c841e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d8c74a3ffbf46ffb8c20b451c841e0a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34171 } } }
I20260926 14:49:07.062831 22600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:07.062830 22599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:07.063246 22607 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:49:07.066125 22607 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:49:07.072356 22607 catalog_manager.cc:1347] Generated new cluster ID: d180921de1ae48959c977023e8bbac7b
I20260926 14:49:07.072425 22607 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:49:07.087989 22607 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:49:07.088373 22607 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:49:07.096481 22607 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d8c74a3ffbf46ffb8c20b451c841e0a: Generated new TSK 0
I20260926 14:49:07.097398 22607 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 14:49:07.102044 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 22520
2026-09-26T14:49:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1342 ms)
[----------] 18 tests from AuthTokenIssuingTest (23858 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:49:07 dist-test-slave-s7k6 krb5kdc[22619](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:49:07 dist-test-slave-s7k6 krb5kdc[22619](info): set up 2 sockets
Sep 26 14:49:07 dist-test-slave-s7k6 krb5kdc[22619](info): commencing operation
krb5kdc: starting...
W20260926 14:49:08.070874 392 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:49:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:49:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:49:08.187639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41957,127.0.98.61:33743,127.0.98.60:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:49:08.300734 22632 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:49:08.301054 22632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:08.301077 22632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:08.301115 22632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:08.305147 22632 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:08.305210 22632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:08.305228 22632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:08.305243 22632 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:08.305256 22632 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:08.310544 22632 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41957,127.0.98.61:33743,127.0.98.60:44297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:08.311646 22632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:08.312637 22632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:49:08.320942 22638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:08.320926 22637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:08.320967 22640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:08.321277 22632 server_base.cc:1034] running on GCE node
I20260926 14:49:08.321892 22632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:08.323132 22632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:08.324306 22632 hybrid_clock.cc:648] HybridClock initialized: now 1790434148324291 us; error 29 us; skew 500 ppm
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:08.327064 22632 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:49:08.327966 22632 webserver.cc:460] Webserver started at http://127.0.98.62:45213/ using document root <none> and password file <none>
I20260926 14:49:08.328504 22632 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:08.328567 22632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:08.328761 22632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:08.330893 22632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "637114ea48be46fcb059092111767e43"
format_stamp: "Formatted at 2026-09-26 14:49:08 on dist-test-slave-s7k6"
server_key: "5fd4f4049f2f8583086dd0e58c86f607"
server_key_iv: "2163fcc5c130bfe339b9473549ff6d6f"
server_key_version: "encryptionkey@0"
I20260926 14:49:08.331430 22632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "637114ea48be46fcb059092111767e43"
format_stamp: "Formatted at 2026-09-26 14:49:08 on dist-test-slave-s7k6"
server_key: "5fd4f4049f2f8583086dd0e58c86f607"
server_key_iv: "2163fcc5c130bfe339b9473549ff6d6f"
server_key_version: "encryptionkey@0"
I20260926 14:49:08.335150 22632 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:49:08.337661 22647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:08.338922 22632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:49:08.339040 22632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "637114ea48be46fcb059092111767e43"
format_stamp: "Formatted at 2026-09-26 14:49:08 on dist-test-slave-s7k6"
server_key: "5fd4f4049f2f8583086dd0e58c86f607"
server_key_iv: "2163fcc5c130bfe339b9473549ff6d6f"
server_key_version: "encryptionkey@0"
I20260926 14:49:08.339143 22632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:08.369547 22632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:08.372824 22632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:08.373096 22632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:08.380035 22632 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41957
I20260926 14:49:08.380286 22699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41957 every 8 connection(s)
I20260926 14:49:08.381024 22632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:49:08.383534 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 22632
I20260926 14:49:08.383664 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:49:08.383972 392 external_mini_cluster.cc:1428] Setting key 75fede2eb505afa92247facfa6acdc2d
I20260926 14:49:08.384140 22700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:08.388338 22700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:49:08.403295 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.390828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.0.1:36488)
0926 14:49:08.391385 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.391440 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:08.392405 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:08.393107 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:08.393113 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:08.393589 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:08.393871 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.393892 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.395099 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.395102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.395812 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.395820 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.396414 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.396418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.396508 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.397091 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:08.397113 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:08.398806 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:08.400610 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.400619 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.400629 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.400853 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.401140 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.401143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.401145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.401255 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.401848 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:08.401855 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:08.401985 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 14:49:08.402171 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 14:49:08.402525 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":43,"thread_start_us":144,"threads_started":1}
I20260926 14:49:08.403301 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.391804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36488 (local address 127.0.98.62:41957)
0926 14:49:08.392115 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.392127 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.392172 (+ 45us) server_negotiation.cc:407] Connection header received
0926 14:49:08.392694 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.392698 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.392892 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.393075 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.394042 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.394976 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.395921 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.396277 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.396417 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.398953 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.398960 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.398970 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.398992 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.400497 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.400938 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.400942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.400944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.401038 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.401327 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.401329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.401331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.401743 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.401843 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.402234 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.402450 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":100,"threads_started":1}
I20260926 14:49:08.405720 22700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:49:08.406435 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.406176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33743 (local address 127.0.0.1:38014)
0926 14:49:08.406299 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.406378 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 14:49:08.406634 22651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33743: connect: Connection refused (error 111)
I20260926 14:49:08.407472 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.407286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33743 (local address 127.0.0.1:38026)
0926 14:49:08.407376 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.407421 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:49:08.407593 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33743: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33743: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 14:49:08.408951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33743
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41957,127.0.98.61:33743,127.0.98.60:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:49:08.456863 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } attempt: 1
I20260926 14:49:08.457765 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.457444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33743 (local address 127.0.0.1:38042)
0926 14:49:08.457613 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.457684 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260926 14:49:08.458519 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.458277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33743 (local address 127.0.0.1:38056)
0926 14:49:08.458408 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.458459 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 14:49:08.458683 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33743: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33743: connect: Connection refused (error 111)
W20260926 14:49:08.519826 22707 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:49:08.520140 22707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:08.520184 22707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:08.520246 22707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:08.524349 22707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:08.524411 22707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:08.524438 22707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:08.524461 22707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:08.524488 22707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:08.526841 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } attempt: 2
I20260926 14:49:08.527661 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.527354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33743 (local address 127.0.0.1:38072)
0926 14:49:08.527521 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.527587 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 14:49:08.528529 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.528304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33743 (local address 127.0.0.1:38084)
0926 14:49:08.528414 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.528464 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:49:08.528642 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:33743: Network error: Client connection negotiation failed: client connection to 127.0.98.61:33743: connect: Connection refused (error 111)
I20260926 14:49:08.530105 22707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41957,127.0.98.61:33743,127.0.98.60:44297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:33743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:08.531312 22707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:08.532362 22707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:49:08.541061 22712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:08.541047 22713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:08.541119 22707 server_base.cc:1034] running on GCE node
W20260926 14:49:08.541052 22715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:08.541815 22707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:08.543056 22707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:08.544225 22707 hybrid_clock.cc:648] HybridClock initialized: now 1790434148544207 us; error 35 us; skew 500 ppm
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:08.546936 22707 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 14:49:08.547859 22707 webserver.cc:460] Webserver started at http://127.0.98.61:46705/ using document root <none> and password file <none>
I20260926 14:49:08.548427 22707 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:08.548493 22707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:08.548688 22707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:08.550717 22707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/data/instance:
uuid: "03479d9b185f4314a7821eacc975da3b"
format_stamp: "Formatted at 2026-09-26 14:49:08 on dist-test-slave-s7k6"
server_key: "b233969b567579eb1cdccc4f97d22280"
server_key_iv: "10e9f24e16489ca1cd3abc9b93600dc9"
server_key_version: "encryptionkey@0"
I20260926 14:49:08.551283 22707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/wal/instance:
uuid: "03479d9b185f4314a7821eacc975da3b"
format_stamp: "Formatted at 2026-09-26 14:49:08 on dist-test-slave-s7k6"
server_key: "b233969b567579eb1cdccc4f97d22280"
server_key_iv: "10e9f24e16489ca1cd3abc9b93600dc9"
server_key_version: "encryptionkey@0"
I20260926 14:49:08.554941 22707 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 14:49:08.557314 22722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:08.558581 22707 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 14:49:08.558699 22707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/wal
uuid: "03479d9b185f4314a7821eacc975da3b"
format_stamp: "Formatted at 2026-09-26 14:49:08 on dist-test-slave-s7k6"
server_key: "b233969b567579eb1cdccc4f97d22280"
server_key_iv: "10e9f24e16489ca1cd3abc9b93600dc9"
server_key_version: "encryptionkey@0"
I20260926 14:49:08.558841 22707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:08.581897 22707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:08.584939 22707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:08.585124 22707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:08.591749 22707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:33743
I20260926 14:49:08.592048 22774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:33743 every 8 connection(s)
I20260926 14:49:08.592749 22707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
I20260926 14:49:08.594878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 22707
I20260926 14:49:08.594982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-1/wal/instance
I20260926 14:49:08.595234 392 external_mini_cluster.cc:1428] Setting key 9819bcb17c5f53c136f6e665bdf808aa
I20260926 14:49:08.595724 22775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:08.599524 22775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:49:08.613235 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.601423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.0.98.62:41957)
0926 14:49:08.601595 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.601600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.602461 (+ 861us) server_negotiation.cc:407] Connection header received
0926 14:49:08.603442 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.603446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.603504 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.603596 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.604866 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.605606 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.607142 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.607382 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.607474 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.609771 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.609774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.609776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.609802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.611161 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.611657 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.611660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.611661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.611737 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.612059 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.612063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.612064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.612260 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.612349 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.612930 (+ 581us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.613064 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:49:08.614116 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.601751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.0.1:36496)
0926 14:49:08.602236 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.602288 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:08.603229 (+ 941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:08.603762 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:08.603768 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:08.604143 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:08.604666 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.604680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.605746 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.605750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.607027 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.607037 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.607527 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.607531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.607690 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.608223 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:08.608239 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:08.609609 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:08.611279 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.611285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.611302 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.611563 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.611817 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.611820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.611822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.611977 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.612368 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:08.612375 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:08.612675 (+ 300us) client_negotiation.cc:769] Sending connection context
0926 14:49:08.612852 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 14:49:08.613220 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":134,"threads_started":1}
I20260926 14:49:08.615670 22775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 14:49:08.619514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44297
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41957,127.0.98.61:33743,127.0.98.60:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:49:08.619789 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } attempt: 3
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 14:49:08.625900 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.616242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33743 (local address 127.0.0.1:38086)
0926 14:49:08.616392 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.616408 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:08.616492 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:08.617165 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:08.617167 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:08.617344 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:08.617425 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.617430 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.618358 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.618362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.618980 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.618989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.619568 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.619571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.619716 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.620735 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:08.620747 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:08.622137 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:08.624200 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.624204 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.624206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.624504 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.624874 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.624878 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.624880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.624976 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.625413 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:08.625416 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:08.625495 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 14:49:08.625578 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 14:49:08.625708 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":34}
I20260926 14:49:08.626159 22780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.616427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.0.98.61:33743)
0926 14:49:08.616779 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.616788 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.616807 (+ 19us) server_negotiation.cc:407] Connection header received
0926 14:49:08.616842 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.616848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.617026 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.617171 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.617521 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.618258 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.619080 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.619466 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.619557 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.622270 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.622280 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.622296 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.622321 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.624095 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.624628 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.624635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.624637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.624758 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.625060 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.625063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.625065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.625310 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.625433 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.625666 (+ 233us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.625795 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":269,"thread_start_us":169,"threads_started":1}
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 14:49:08.628429 22775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:49:08.629096 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.628851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49070)
0926 14:49:08.628957 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.629031 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 14:49:08.629333 22724 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
I20260926 14:49:08.630120 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.629932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49076)
0926 14:49:08.630023 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.630068 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 14:49:08.630353 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.620196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33743 (local address 127.0.0.1:38096)
0926 14:49:08.620433 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.620444 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:08.620511 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:08.621332 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:08.621336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:08.621510 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:08.621641 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.621648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.623009 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.623013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.623854 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.623867 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.624431 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.624435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.624541 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.625012 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:08.625028 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:08.626564 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:08.628583 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.628587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.628589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.628842 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.629162 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.629166 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.629168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.629285 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.629823 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:08.629826 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:08.629941 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 14:49:08.630028 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 14:49:08.630172 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167}
W20260926 14:49:08.630328 22775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44297: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
I20260926 14:49:08.631079 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.620440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.0.98.61:33743)
0926 14:49:08.621049 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.621057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.621071 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:49:08.621105 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.621107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.621148 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.621279 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.621782 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.622519 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.623987 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.624310 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.624403 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.626726 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.626730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.626732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.626780 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.628476 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.628964 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.628967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.628969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.629053 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.629420 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.629423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.629425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.629675 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.630678 (+ 1003us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.630787 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.630904 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":95,"threads_started":1}
I20260926 14:49:08.631681 22700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:49:08.632373 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.632126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49082)
0926 14:49:08.632238 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.632295 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 14:49:08.633164 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.632918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49098)
0926 14:49:08.633062 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.633111 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260926 14:49:08.633299 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44297: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
I20260926 14:49:08.664499 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } attempt: 1
I20260926 14:49:08.665320 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.665026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49110)
0926 14:49:08.665179 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.665240 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 14:49:08.666074 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.665856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49118)
0926 14:49:08.665969 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.666014 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 14:49:08.666200 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44297: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
I20260926 14:49:08.679579 22775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } attempt: 1
I20260926 14:49:08.680439 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.680139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49128)
0926 14:49:08.680279 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.680361 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 14:49:08.681077 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.680898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49140)
0926 14:49:08.680980 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.681024 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 14:49:08.681190 22775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44297: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
W20260926 14:49:08.733846 22782 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:49:08.734124 22782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:08.734146 22782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:08.734195 22782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:08.738256 22782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:08.738343 22782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:08.738363 22782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:08.738377 22782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:08.738395 22782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:08.741384 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } attempt: 2
I20260926 14:49:08.742588 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.742288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49142)
0926 14:49:08.742435 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.742504 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 14:49:08.743337 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.743136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49154)
0926 14:49:08.743238 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.743284 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:49:08.743468 22700 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44297: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
I20260926 14:49:08.744489 22782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41957,127.0.98.61:33743,127.0.98.60:44297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:08.745581 22782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:08.746579 22782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 14:49:08.749326 22775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } attempt: 2
I20260926 14:49:08.749982 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.749713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49164)
0926 14:49:08.749864 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.749916 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 14:49:08.750859 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.750554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49180)
0926 14:49:08.750702 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.750779 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260926 14:49:08.751005 22775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:44297: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44297: connect: Connection refused (error 111)
W20260926 14:49:08.754606 22791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:08.754611 22788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:08.754877 22782 server_base.cc:1034] running on GCE node
W20260926 14:49:08.754664 22789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:08.755541 22782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:08.756609 22782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:08.757783 22782 hybrid_clock.cc:648] HybridClock initialized: now 1790434148757755 us; error 39 us; skew 500 ppm
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:08.760388 22782 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 14:49:08.761224 22782 webserver.cc:460] Webserver started at http://127.0.98.60:39839/ using document root <none> and password file <none>
I20260926 14:49:08.761741 22782 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:08.761806 22782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:08.762024 22782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:08.764127 22782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/data/instance:
uuid: "4712752e17b4419faa76b9f30d958d68"
format_stamp: "Formatted at 2026-09-26 14:49:08 on dist-test-slave-s7k6"
server_key: "b6449f4d080921e101aa3e7be52b0b45"
server_key_iv: "b32f53b2d0e258862de3c617ed027a4c"
server_key_version: "encryptionkey@0"
I20260926 14:49:08.764654 22782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/wal/instance:
uuid: "4712752e17b4419faa76b9f30d958d68"
format_stamp: "Formatted at 2026-09-26 14:49:08 on dist-test-slave-s7k6"
server_key: "b6449f4d080921e101aa3e7be52b0b45"
server_key_iv: "b32f53b2d0e258862de3c617ed027a4c"
server_key_version: "encryptionkey@0"
I20260926 14:49:08.768194 22782 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 14:49:08.770522 22798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:08.771761 22782 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 14:49:08.771873 22782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/wal
uuid: "4712752e17b4419faa76b9f30d958d68"
format_stamp: "Formatted at 2026-09-26 14:49:08 on dist-test-slave-s7k6"
server_key: "b6449f4d080921e101aa3e7be52b0b45"
server_key_iv: "b32f53b2d0e258862de3c617ed027a4c"
server_key_version: "encryptionkey@0"
I20260926 14:49:08.771967 22782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:08.805393 22782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:08.808607 22782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:08.808790 22782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:08.815635 22782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44297
I20260926 14:49:08.815912 22850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44297 every 8 connection(s)
I20260926 14:49:08.816681 22782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
I20260926 14:49:08.819600 22851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:08.823431 22851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:49:08.826733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 22782
I20260926 14:49:08.826856 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790433872429009-392-0/minicluster-data/master-2/wal/instance
I20260926 14:49:08.827116 392 external_mini_cluster.cc:1428] Setting key 9c6eb56722230bcb2b801451cf01216f
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 14:49:08.836988 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.825206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36508 (local address 127.0.98.62:41957)
0926 14:49:08.825355 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.825360 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.826202 (+ 842us) server_negotiation.cc:407] Connection header received
0926 14:49:08.827192 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.827196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.827254 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.827351 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.828630 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.829175 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.830558 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.830803 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.830893 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.833530 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.833533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.833535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.833560 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.834984 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.835582 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.835584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.835586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.835653 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.835959 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.835962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.835964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.836141 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.836222 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.836714 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.836828 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 14:49:08.837812 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.825612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41957 (local address 127.0.0.1:36508)
0926 14:49:08.826054 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.826097 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:08.826978 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:08.827504 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:08.827513 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:08.827992 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:08.828498 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.828510 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.829287 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.829289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.830453 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.830462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.830926 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.830930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.831145 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.831772 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:08.831791 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:08.833388 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:08.835080 (+ 1692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.835087 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.835097 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.835495 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.835742 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.835745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.835747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.835875 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.836233 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:08.836240 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:08.836491 (+ 251us) client_negotiation.cc:769] Sending connection context
0926 14:49:08.836658 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 14:49:08.836948 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260926 14:49:08.837817 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.827746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49196 (local address 127.0.98.60:44297)
0926 14:49:08.828008 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.828018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.828260 (+ 242us) server_negotiation.cc:407] Connection header received
0926 14:49:08.828369 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.828375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.828532 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.828675 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.829366 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.830186 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.831052 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.831479 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.831551 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.833888 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.833893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.833902 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.833921 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.835585 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.835959 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.835965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.835966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.836040 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.836277 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.836281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.836282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.836554 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.836754 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.836953 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.837080 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
I20260926 14:49:08.839032 22851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:49:08.842175 22775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } attempt: 3
I20260926 14:49:08.842619 22700 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } attempt: 3
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 14:49:08.848776 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.839430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:33743 (local address 127.0.0.1:38110)
0926 14:49:08.839666 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.839688 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:08.839759 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:08.840249 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:08.840252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:08.840408 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:08.840512 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.840518 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.841337 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.841340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.841969 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.841978 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.842514 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.842518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.842875 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.843492 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:08.843505 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:08.845452 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:08.847304 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.847307 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.847309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.847501 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.847752 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.847755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.847757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.847840 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.848280 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:08.848282 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:08.848372 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 14:49:08.848477 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 14:49:08.848593 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260926 14:49:08.848831 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.839553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38110 (local address 127.0.98.61:33743)
0926 14:49:08.839677 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.839681 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.839730 (+ 49us) server_negotiation.cc:407] Connection header received
0926 14:49:08.839830 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.839833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.839882 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.839961 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.840622 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.841239 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.842077 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.842427 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.842515 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.845579 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.845583 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.845585 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.845611 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.847192 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.847586 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.847590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.847592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.847666 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.847945 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.847948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.847950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.848138 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.848210 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.848560 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.848675 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 14:49:08.849586 22851 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:49:08.850641 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.840166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36520 (local address 127.0.98.62:41957)
0926 14:49:08.840274 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.840277 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.840287 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:49:08.840380 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.840381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.840414 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.840468 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.841017 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.841579 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.842465 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.843083 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.843171 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.847014 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.847017 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.847020 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.847045 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.848857 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.849277 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.849281 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.849282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.849354 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.849645 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.849648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.849650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.849863 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.850067 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.850402 (+ 335us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.850504 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:49:08.853421 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.842555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49202)
0926 14:49:08.842681 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.842694 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:08.842778 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:08.843339 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:08.843342 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:08.843511 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:08.843599 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.843606 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.845613 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.845616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.846410 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.846420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.847032 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.847035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.847119 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.847515 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:08.847530 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:08.849768 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:08.851964 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.851967 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.851969 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.852202 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.852485 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.852488 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.852489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.852570 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.852970 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:08.852973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:08.853056 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 14:49:08.853141 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 14:49:08.853267 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 14:49:08.853431 22703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.842994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49218)
0926 14:49:08.843145 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.843157 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:08.843225 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:08.844014 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:08.844017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:08.844166 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:08.844243 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.844248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.845366 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.845370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.845949 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.845958 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.846606 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.846610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.846691 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.847115 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:08.847130 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:08.849280 (+ 2150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:08.851906 (+ 2626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.851909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.851911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.852140 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.852440 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.852443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.852446 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.852528 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.852968 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:08.852971 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:08.853067 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 14:49:08.853161 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 14:49:08.853284 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 14:49:08.854017 22856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.843419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49218 (local address 127.0.98.60:44297)
0926 14:49:08.843817 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.843825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.843834 (+ 9us) server_negotiation.cc:407] Connection header received
0926 14:49:08.843858 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.843860 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.843888 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.843999 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.844342 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.845249 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.846053 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.846289 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.846376 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.849407 (+ 3031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.849410 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.849413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.849434 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.851801 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.852245 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.852249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.852250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.852346 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.852622 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.852625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.852626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.852834 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.852923 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.853725 (+ 802us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.853832 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"spinlock_wait_cycles":2816,"thread_start_us":103,"threads_started":1}
I20260926 14:49:08.854466 22854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.842812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.0.98.60:44297)
0926 14:49:08.842908 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.842912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.842922 (+ 10us) server_negotiation.cc:407] Connection header received
0926 14:49:08.842949 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.842952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.842985 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.843311 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.843722 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.845148 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.846491 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.846835 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.846925 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.849955 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.849960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.849963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.849986 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.851853 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.852304 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.852307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.852309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.852394 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.852664 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.852667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.852669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.852854 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.852930 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.854034 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.854324 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":43}
I20260926 14:49:08.857344 22775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b: Bootstrap starting.
I20260926 14:49:08.857407 22700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43: Bootstrap starting.
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 14:49:08 dist-test-slave-s7k6 krb5kdc[22619](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 14:49:08.859975 22700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:08.859975 22775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:08.860836 22775 log.cc:824] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:08.860834 22700 log.cc:824] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:08.862241 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.852067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.0.98.61:33743)
0926 14:49:08.852164 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.852167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.852322 (+ 155us) server_negotiation.cc:407] Connection header received
0926 14:49:08.852475 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.852478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.852518 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.852565 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.853346 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.853990 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.855602 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.855893 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.855989 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.858846 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.858850 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.858853 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.858877 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.860415 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.860908 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.860912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.860914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.860992 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.861324 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.861328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.861330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.861559 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.861768 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.861958 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.862083 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 14:49:08.863593 22700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43: No bootstrap required, opened a new log
I20260926 14:49:08.863593 22775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b: No bootstrap required, opened a new log
I20260926 14:49:08.864523 22853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.850252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44297 (local address 127.0.0.1:49220)
0926 14:49:08.851019 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:08.851031 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:08.851096 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:08.852076 (+ 980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:08.852078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:08.852210 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:08.852299 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.852304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.853204 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.853206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.853954 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:08.853964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.854509 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.854513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:08.854618 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.858883 (+ 4265us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:08.858897 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:08.860351 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:08.862956 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.862958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.862960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.863181 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.863498 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:08.863501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:08.863503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:08.863582 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:08.864049 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:08.864051 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:08.864135 (+ 84us) client_negotiation.cc:769] Sending connection context
0926 14:49:08.864233 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 14:49:08.864366 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":19}
I20260926 14:49:08.865281 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.850449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.0.98.60:44297)
0926 14:49:08.851807 (+ 1358us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.851819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.851832 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:49:08.851860 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.851863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.851900 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.851968 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.852392 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.853115 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.854055 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.854404 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.854503 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.860494 (+ 5991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.860497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.860499 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.860521 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.862856 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.863284 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.863287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.863289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.863365 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.863721 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.863724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.863726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.863933 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.864017 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.864995 (+ 978us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.865112 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1294,"thread_start_us":309,"threads_started":1}
I20260926 14:49:08.865712 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:49:08.866942 22674 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "637114ea48be46fcb059092111767e43"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36520
I20260926 14:49:08.867684 22775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } }
I20260926 14:49:08.867684 22700 raft_consensus.cc:348] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } }
I20260926 14:49:08.867908 22700 raft_consensus.cc:729] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 637114ea48be46fcb059092111767e43, State: Initialized, Role: FOLLOWER
I20260926 14:49:08.867908 22775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03479d9b185f4314a7821eacc975da3b, State: Initialized, Role: FOLLOWER
I20260926 14:49:08.868248 22700 consensus_queue.cc:260] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } }
I20260926 14:49:08.868248 22775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } }
I20260926 14:49:08.869052 22851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68: Bootstrap starting.
I20260926 14:49:08.869004 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } } }
I20260926 14:49:08.869637 22775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:08.869637 22700 sys_catalog.cc:564] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:08.869736 22674 raft_consensus.cc:482] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 14:49:08.869843 22674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:49:08.871053 22674 raft_consensus.cc:504] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } }
I20260926 14:49:08.872023 22674 leader_election.cc:288] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [CANDIDATE]: Term 1 election: Requested vote from peers 03479d9b185f4314a7821eacc975da3b (127.0.98.61:33743), 4712752e17b4419faa76b9f30d958d68 (127.0.98.60:44297)
I20260926 14:49:08.872463 22749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "637114ea48be46fcb059092111767e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "03479d9b185f4314a7821eacc975da3b"
I20260926 14:49:08.873250 22749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:49:08.874537 22749 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 637114ea48be46fcb059092111767e43 in term 1.
I20260926 14:49:08.874984 22651 leader_election.cc:302] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03479d9b185f4314a7821eacc975da3b, 637114ea48be46fcb059092111767e43; no voters:
W20260926 14:49:08.875403 22653 tablet.cc:2369] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 14:49:08.875413 22882 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:49:08.875471 22882 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:49:08.875420 22889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:49:08.875862 22889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [term 1 LEADER]: Becoming Leader. State: Replica: 637114ea48be46fcb059092111767e43, State: Running, Role: LEADER
I20260926 14:49:08.876322 22889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } }
I20260926 14:49:08.878077 22889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 637114ea48be46fcb059092111767e43. Latest consensus state: current_term: 1 leader_uuid: "637114ea48be46fcb059092111767e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } } }
I20260926 14:49:08.878239 22889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:08.878592 22851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:08.879884 22851 log.cc:824] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:08.869643 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:08.869047 22861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } } }
I20260926 14:49:08.880321 22861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [sys.catalog]: This master's current role is: FOLLOWER
W20260926 14:49:08.880896 22883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:49:08.880959 22883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:49:08.882109 22851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68: No bootstrap required, opened a new log
I20260926 14:49:08.879683 22632 master_runner.cc:186] Error getting master registration for 127.0.98.60:44297: OK, instance_id { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" instance_seqno: 1790434148813821 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:49:08.883411 22632 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:49:08.887112 22851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } }
I20260926 14:49:08.887347 22851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4712752e17b4419faa76b9f30d958d68, State: Initialized, Role: FOLLOWER
I20260926 14:49:08.887678 22851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } }
I20260926 14:49:08.884922 22707 master_runner.cc:186] Error getting master registration for 127.0.98.60:44297: OK, instance_id { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" instance_seqno: 1790434148813821 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:49:08.888232 22707 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:49:08.888912 22851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:08.888998 22825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "637114ea48be46fcb059092111767e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4712752e17b4419faa76b9f30d958d68"
I20260926 14:49:08.889233 22825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:49:08.890529 22825 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 637114ea48be46fcb059092111767e43 in term 1.
I20260926 14:49:08.892920 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } } }
I20260926 14:49:08.893050 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:08.900534 22892 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:49:08.901629 22892 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 14:49:08.904458 22704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.891146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.0.98.62:41957)
0926 14:49:08.891261 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.891265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.891712 (+ 447us) server_negotiation.cc:407] Connection header received
0926 14:49:08.891911 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.891914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.891963 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.892014 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.893031 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.893746 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.894922 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.895272 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.895340 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.899494 (+ 4154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.899497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.899500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.899526 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.901202 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.902697 (+ 1495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.902700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.902702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.902815 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.903107 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.903111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.903113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.903358 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.903457 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.904179 (+ 722us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.904305 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 14:49:08.906967 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.891834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.0.98.60:44297)
0926 14:49:08.891936 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.891940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.893398 (+ 1458us) server_negotiation.cc:407] Connection header received
0926 14:49:08.893545 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.893548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.893592 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.893637 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.894619 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.895386 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.896442 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.896785 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.896854 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.903784 (+ 6930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.903787 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.903790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.903818 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.905352 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.905805 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.905808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.905810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.905896 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.906175 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.906178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.906181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.906351 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.906483 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.906680 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.906819 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 14:49:08.909708 22909 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:49:08.909791 22909 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:49:08.912678 22781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:08.892529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.0.98.61:33743)
0926 14:49:08.892657 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:08.892661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:08.903051 (+ 10390us) server_negotiation.cc:407] Connection header received
0926 14:49:08.903238 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:08.903241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:08.903290 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:08.903337 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:08.904127 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.904893 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.906019 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:08.906345 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:08.906413 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:08.909242 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:08.909246 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:08.909248 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:08.909275 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:08.910847 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.911318 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.911321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.911323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.911421 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:08.911728 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:08.911732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:08.911734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:08.911945 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:08.912042 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:08.912397 (+ 355us) server_negotiation.cc:299] Negotiation successful
0926 14:49:08.912513 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 14:49:08.922480 22825 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [term 1 FOLLOWER]: Refusing update from remote peer 637114ea48be46fcb059092111767e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:49:08.923146 22860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:49:08.924072 22749 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [term 1 FOLLOWER]: Refusing update from remote peer 637114ea48be46fcb059092111767e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:49:08.924600 22860 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:49:08.927054 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 637114ea48be46fcb059092111767e43. Latest consensus state: current_term: 1 leader_uuid: "637114ea48be46fcb059092111767e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } } }
I20260926 14:49:08.927191 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:08.927847 22861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 637114ea48be46fcb059092111767e43. Latest consensus state: current_term: 1 leader_uuid: "637114ea48be46fcb059092111767e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } } }
I20260926 14:49:08.927966 22861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:08.929819 22860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "637114ea48be46fcb059092111767e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } } }
I20260926 14:49:08.929944 22860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:08.930974 22895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "637114ea48be46fcb059092111767e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } } }
I20260926 14:49:08.931082 22895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:08.932340 22910 mvcc.cc:205] Tried to move back new op lower bound from 7333618273979052032 to 7333618273801003008. Current Snapshot: MvccSnapshot[applied={T|T < 7333618273979052032}]
I20260926 14:49:08.933213 22912 mvcc.cc:205] Tried to move back new op lower bound from 7333618273979052032 to 7333618273801003008. Current Snapshot: MvccSnapshot[applied={T|T < 7333618273979052032}]
I20260926 14:49:08.942153 22890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "637114ea48be46fcb059092111767e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } } }
I20260926 14:49:08.942310 22890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:08.943284 22861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "637114ea48be46fcb059092111767e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "637114ea48be46fcb059092111767e43" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41957 } } peers { permanent_uuid: "03479d9b185f4314a7821eacc975da3b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 33743 } } peers { permanent_uuid: "4712752e17b4419faa76b9f30d958d68" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44297 } } }
I20260926 14:49:08.943405 22861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:08.943526 22914 mvcc.cc:205] Tried to move back new op lower bound from 7333618273979052032 to 7333618273801003008. Current Snapshot: MvccSnapshot[applied={T|T < 7333618273979052032}]
I20260926 14:49:08.943763 22892 catalog_manager.cc:1347] Generated new cluster ID: 629fe7ed7a2742fd831d77250183f4bd
I20260926 14:49:08.943825 22892 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:49:08.987082 22892 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:49:08.988464 22892 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:49:09.009290 22892 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 637114ea48be46fcb059092111767e43: Generated new TSK 0
I20260926 14:49:09.012665 22892 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:49:09.053994 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.051231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.0.98.61:33743)
0926 14:49:09.051380 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.051385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.052766 (+ 1381us) server_negotiation.cc:407] Connection header received
0926 14:49:09.052997 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.053002 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.053059 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.053133 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.053842 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38150: BlockingRecv error: recv got EOF from 127.0.0.1:38150 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 14:49:09.055657 22857 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.052178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49236 (local address 127.0.98.60:44297)
0926 14:49:09.052294 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.052299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.054576 (+ 2277us) server_negotiation.cc:407] Connection header received
0926 14:49:09.054790 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.054794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.054850 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.054909 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.055514 (+ 605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49236: BlockingRecv error: recv got EOF from 127.0.0.1:49236 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 14:49:09.057806 22704 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.056204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36534 (local address 127.0.98.62:41957)
0926 14:49:09.056331 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.056336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.056743 (+ 407us) server_negotiation.cc:407] Connection header received
0926 14:49:09.057038 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.057041 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.057107 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.057158 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.057669 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36534: BlockingRecv error: recv got EOF from 127.0.0.1:36534 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 14:49:09.062970 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.061035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.0.98.61:33743)
0926 14:49:09.061160 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.061165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.061723 (+ 558us) server_negotiation.cc:407] Connection header received
0926 14:49:09.061949 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.061953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.062047 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.062118 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.062853 (+ 735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38166: BlockingRecv error: recv got EOF from 127.0.0.1:38166 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 14:49:09.068349 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.066674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.0.98.61:33743)
0926 14:49:09.066827 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.066832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.067421 (+ 589us) server_negotiation.cc:407] Connection header received
0926 14:49:09.067612 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.067615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.067667 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.067719 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.068246 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38180: BlockingRecv error: recv got EOF from 127.0.0.1:38180 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 14:49:09.073760 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.072364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.0.98.61:33743)
0926 14:49:09.072458 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.072462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.072900 (+ 438us) server_negotiation.cc:407] Connection header received
0926 14:49:09.073065 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.073068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.073127 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.073175 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.073654 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38190: BlockingRecv error: recv got EOF from 127.0.0.1:38190 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 14:49:09.081666 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.080003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38198 (local address 127.0.98.61:33743)
0926 14:49:09.080159 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.080163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.080704 (+ 541us) server_negotiation.cc:407] Connection header received
0926 14:49:09.080884 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.080887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.080936 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.080983 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.081549 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38198: BlockingRecv error: recv got EOF from 127.0.0.1:38198 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260926 14:49:09.093798 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.092022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.0.98.61:33743)
0926 14:49:09.092152 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.092157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.092770 (+ 613us) server_negotiation.cc:407] Connection header received
0926 14:49:09.092966 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.092969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.093021 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.093082 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.093672 (+ 590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38208: BlockingRecv error: recv got EOF from 127.0.0.1:38208 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 14:49:09.113166 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.112172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38224 (local address 127.0.98.61:33743)
0926 14:49:09.112287 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.112291 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.112445 (+ 154us) server_negotiation.cc:407] Connection header received
0926 14:49:09.112565 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.112568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.112614 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.112696 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.113066 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38224: BlockingRecv error: recv got EOF from 127.0.0.1:38224 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 14:49:09.148402 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.147233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38238 (local address 127.0.98.61:33743)
0926 14:49:09.147385 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.147389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.147636 (+ 247us) server_negotiation.cc:407] Connection header received
0926 14:49:09.147761 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.147764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.147812 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.147878 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.148271 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38238: BlockingRecv error: recv got EOF from 127.0.0.1:38238 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 14:49:09.215842 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.214657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38242 (local address 127.0.98.61:33743)
0926 14:49:09.214845 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.214849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.215002 (+ 153us) server_negotiation.cc:407] Connection header received
0926 14:49:09.215101 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.215104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.215160 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.215246 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.215731 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38242: BlockingRecv error: recv got EOF from 127.0.0.1:38242 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 14:49:09.347204 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.346039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38252 (local address 127.0.98.61:33743)
0926 14:49:09.346174 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.346178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.346276 (+ 98us) server_negotiation.cc:407] Connection header received
0926 14:49:09.346462 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.346465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.346519 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.346603 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.347096 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38252: BlockingRecv error: recv got EOF from 127.0.0.1:38252 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 14:49:09.606328 22781 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:09.605368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.0.98.61:33743)
0926 14:49:09.605508 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:09.605511 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:09.605580 (+ 69us) server_negotiation.cc:407] Connection header received
0926 14:49:09.605698 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:09.605701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:09.605744 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:09.605814 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:09.606217 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38254: BlockingRecv error: recv got EOF from 127.0.0.1:38254 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 14:49:09.876848 22882 catalog_manager.cc:1259] Loaded cluster ID: 629fe7ed7a2742fd831d77250183f4bd
I20260926 14:49:09.876929 22882 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b: loading cluster ID for follower catalog manager: success
I20260926 14:49:09.878638 22882 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b: acquiring CA information for follower catalog manager: success
I20260926 14:49:09.879853 22882 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 03479d9b185f4314a7821eacc975da3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:49:09.911069 22909 catalog_manager.cc:1259] Loaded cluster ID: 629fe7ed7a2742fd831d77250183f4bd
I20260926 14:49:09.911150 22909 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68: loading cluster ID for follower catalog manager: success
I20260926 14:49:09.912884 22909 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68: acquiring CA information for follower catalog manager: success
I20260926 14:49:09.914065 22909 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4712752e17b4419faa76b9f30d958d68: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:49:10.124703 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:10.120789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.0.98.61:33743)
0926 14:49:10.121014 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:10.121018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:10.121092 (+ 74us) server_negotiation.cc:407] Connection header received
0926 14:49:10.121258 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:10.121261 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:10.121388 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:10.121496 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:49:10.121723 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:10.122262 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:10.123280 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:10.123499 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:10.123571 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:10.123866 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:49:10.124317 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:49:10.124408 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:10.124517 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 14:49:10.124572 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":76,"threads_started":1}
I20260926 14:49:10.131604 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:10.127730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.0.98.60:44297)
0926 14:49:10.127950 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:10.127954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:10.128058 (+ 104us) server_negotiation.cc:407] Connection header received
0926 14:49:10.128156 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:10.128158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:10.128275 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:10.128374 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:49:10.128621 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:10.129146 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:10.130036 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:10.130354 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:10.130446 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:10.130690 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:49:10.131206 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:49:10.131304 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:10.131386 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 14:49:10.131442 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":66,"threads_started":1}
I20260926 14:49:10.138710 22988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:10.134894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49258 (local address 127.0.98.60:44297)
0926 14:49:10.135006 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:10.135010 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:10.135320 (+ 310us) server_negotiation.cc:407] Connection header received
0926 14:49:10.135429 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:10.135432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:10.135482 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:10.135549 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:49:10.135783 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:10.136396 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:10.137488 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:10.137842 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:10.137953 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:10.138182 (+ 229us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:49:10.138263 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:49:10.138336 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:10.138505 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 14:49:10.138563 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 14:49:10.139236 22982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:10.134727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.0.98.61:33743)
0926 14:49:10.134880 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:10.134884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:10.135371 (+ 487us) server_negotiation.cc:407] Connection header received
0926 14:49:10.135554 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:10.135558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:10.135610 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:10.135671 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:49:10.136031 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:10.136893 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:10.138012 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:10.138290 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:10.138361 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:10.138728 (+ 367us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:49:10.138826 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:49:10.138870 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:10.139034 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 14:49:10.139098 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 14:49:10.139350 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:10.134691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36548 (local address 127.0.98.62:41957)
0926 14:49:10.134978 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:10.134982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:10.135152 (+ 170us) server_negotiation.cc:407] Connection header received
0926 14:49:10.135369 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:10.135373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:10.135536 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:10.135656 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:49:10.135862 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:10.136749 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:10.137828 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:10.138127 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:10.138223 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:10.138509 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:49:10.138918 (+ 409us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:49:10.138997 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:10.139145 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 14:49:10.139212 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":113,"threads_started":1}
I20260926 14:49:10.150205 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 22632
I20260926 14:49:10.156059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 22707
I20260926 14:49:10.161435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 22782
2026-09-26T14:49:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3061 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 14:49:10 dist-test-slave-s7k6 krb5kdc[22999](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 14:49:10 dist-test-slave-s7k6 krb5kdc[22999](info): set up 2 sockets
Sep 26 14:49:10 dist-test-slave-s7k6 krb5kdc[22999](info): commencing operation
krb5kdc: starting...
W20260926 14:49:11.130920 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/test-user.keytab.
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T14:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T14:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 14:49:11.247416 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37787
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37787,127.0.98.61:42721,127.0.98.60:33065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 14:49:11.357335 23012 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:49:11.357630 23012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:11.357654 23012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:11.357690 23012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:11.361697 23012 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:11.361758 23012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:11.361778 23012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:11.361791 23012 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:11.361805 23012 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:11.367105 23012 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37787,127.0.98.61:42721,127.0.98.60:33065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:11.368106 23012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:11.369096 23012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:49:11.377444 23017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:11.377581 23020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:11.377444 23018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:11.377750 23012 server_base.cc:1034] running on GCE node
I20260926 14:49:11.378438 23012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:11.379694 23012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:11.380882 23012 hybrid_clock.cc:648] HybridClock initialized: now 1790434151380867 us; error 30 us; skew 500 ppm
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:11.383883 23012 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 14:49:11.384840 23012 webserver.cc:460] Webserver started at http://127.0.98.62:37293/ using document root <none> and password file <none>
I20260926 14:49:11.385385 23012 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:11.385452 23012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:11.385654 23012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:11.387822 23012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/data/instance:
uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55"
format_stamp: "Formatted at 2026-09-26 14:49:11 on dist-test-slave-s7k6"
server_key: "5075f019a203e4ed8820e37a636ebe5a"
server_key_iv: "8313757ddd2e6cdef105984c28605341"
server_key_version: "encryptionkey@0"
I20260926 14:49:11.388376 23012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/wal/instance:
uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55"
format_stamp: "Formatted at 2026-09-26 14:49:11 on dist-test-slave-s7k6"
server_key: "5075f019a203e4ed8820e37a636ebe5a"
server_key_iv: "8313757ddd2e6cdef105984c28605341"
server_key_version: "encryptionkey@0"
I20260926 14:49:11.392160 23012 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 14:49:11.394721 23027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:11.396047 23012 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 14:49:11.396170 23012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/wal
uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55"
format_stamp: "Formatted at 2026-09-26 14:49:11 on dist-test-slave-s7k6"
server_key: "5075f019a203e4ed8820e37a636ebe5a"
server_key_iv: "8313757ddd2e6cdef105984c28605341"
server_key_version: "encryptionkey@0"
I20260926 14:49:11.396271 23012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:11.407328 23012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:11.410437 23012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:11.410625 23012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:11.417718 23012 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37787
I20260926 14:49:11.417990 23079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37787 every 8 connection(s)
I20260926 14:49:11.418769 23012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/data/info.pb
I20260926 14:49:11.421562 23080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:11.423383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 23012
I20260926 14:49:11.423491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-0/wal/instance
I20260926 14:49:11.423763 392 external_mini_cluster.cc:1428] Setting key 7a5fda338829cec7a20ac95049449470
I20260926 14:49:11.425591 23080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:49:11.441742 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.428341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37787 (local address 127.0.0.1:56650)
0926 14:49:11.429065 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.429116 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:11.429998 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:11.430863 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:11.430868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:11.431193 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:11.431493 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.431509 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.432680 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.432682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.433408 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.433417 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.434152 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.434157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.434281 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.434837 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:11.434856 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:11.436675 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:11.438923 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.438931 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.438939 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.439179 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.439526 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.439528 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.439529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.439629 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.440356 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:11.440361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:11.440488 (+ 127us) client_negotiation.cc:769] Sending connection context
0926 14:49:11.440631 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 14:49:11.440914 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"mutex_wait_us":4,"thread_start_us":144,"threads_started":1}
I20260926 14:49:11.441744 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.428341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.0.98.62:37787)
0926 14:49:11.428840 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.428864 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.429251 (+ 387us) server_negotiation.cc:407] Connection header received
0926 14:49:11.430324 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.430331 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.430608 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.430849 (+ 241us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.431649 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.432568 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.433538 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.433980 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.434115 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.436826 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.436833 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.436847 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.436873 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.438815 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.439286 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.439292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.439295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.439441 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.439713 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.439716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.439718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.440251 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.440394 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.440739 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.440992 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":329,"spinlock_wait_cycles":75648,"thread_start_us":144,"threads_started":1}
I20260926 14:49:11.444207 23080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:49:11.444976 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.444745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42721 (local address 127.0.0.1:36252)
0926 14:49:11.444847 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.444915 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:49:11.445163 23031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42721: connect: Connection refused (error 111)
I20260926 14:49:11.445914 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.445726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42721 (local address 127.0.0.1:36266)
0926 14:49:11.445824 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.445865 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 14:49:11.446028 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42721: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42721: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 14:49:11.450614 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42721
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37787,127.0.98.61:42721,127.0.98.60:33065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:49:11.495265 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } attempt: 1
I20260926 14:49:11.496165 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.495862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42721 (local address 127.0.0.1:36270)
0926 14:49:11.496018 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.496083 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 14:49:11.496834 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.496636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42721 (local address 127.0.0.1:36284)
0926 14:49:11.496732 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.496776 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 14:49:11.496946 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42721: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42721: connect: Connection refused (error 111)
W20260926 14:49:11.559015 23087 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:49:11.559301 23087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:11.559340 23087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:11.559402 23087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:11.563449 23087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:11.563522 23087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:11.563548 23087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:11.563570 23087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:11.563589 23087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:11.565109 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } attempt: 2
I20260926 14:49:11.565863 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.565568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42721 (local address 127.0.0.1:36290)
0926 14:49:11.565726 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.565787 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 14:49:11.566540 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.566332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42721 (local address 127.0.0.1:36292)
0926 14:49:11.566434 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.566482 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 14:49:11.566670 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42721: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42721: connect: Connection refused (error 111)
I20260926 14:49:11.569242 23087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37787,127.0.98.61:42721,127.0.98.60:33065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:11.570326 23087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:11.571381 23087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:49:11.579643 23095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:11.579653 23093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:11.579667 23092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:11.579826 23087 server_base.cc:1034] running on GCE node
I20260926 14:49:11.580468 23087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:11.581818 23087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:11.582989 23087 hybrid_clock.cc:648] HybridClock initialized: now 1790434151582966 us; error 41 us; skew 500 ppm
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:11.585768 23087 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 14:49:11.586630 23087 webserver.cc:460] Webserver started at http://127.0.98.61:35959/ using document root <none> and password file <none>
I20260926 14:49:11.587194 23087 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:11.587256 23087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:11.587424 23087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:11.589457 23087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/data/instance:
uuid: "9af81368bcdc429a8724963af7c49cc9"
format_stamp: "Formatted at 2026-09-26 14:49:11 on dist-test-slave-s7k6"
server_key: "189f54c6234f7de372db7fbed30c3c0e"
server_key_iv: "a4a2ce17b4d4dfdd0af4112c4a516097"
server_key_version: "encryptionkey@0"
I20260926 14:49:11.589975 23087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/wal/instance:
uuid: "9af81368bcdc429a8724963af7c49cc9"
format_stamp: "Formatted at 2026-09-26 14:49:11 on dist-test-slave-s7k6"
server_key: "189f54c6234f7de372db7fbed30c3c0e"
server_key_iv: "a4a2ce17b4d4dfdd0af4112c4a516097"
server_key_version: "encryptionkey@0"
I20260926 14:49:11.593698 23087 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 14:49:11.596179 23102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:11.597529 23087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 14:49:11.597644 23087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/wal
uuid: "9af81368bcdc429a8724963af7c49cc9"
format_stamp: "Formatted at 2026-09-26 14:49:11 on dist-test-slave-s7k6"
server_key: "189f54c6234f7de372db7fbed30c3c0e"
server_key_iv: "a4a2ce17b4d4dfdd0af4112c4a516097"
server_key_version: "encryptionkey@0"
I20260926 14:49:11.597759 23087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:11.615690 23087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:11.618836 23087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:11.619027 23087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:11.625952 23087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42721
I20260926 14:49:11.626260 23154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42721 every 8 connection(s)
I20260926 14:49:11.626959 23087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/data/info.pb
I20260926 14:49:11.629776 23155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:11.633484 23155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:49:11.636583 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 23087
I20260926 14:49:11.636705 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-1/wal/instance
I20260926 14:49:11.636986 392 external_mini_cluster.cc:1428] Setting key 32b57eec096557c958f15594f9261624
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 14:49:11.646627 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.635354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.0.98.62:37787)
0926 14:49:11.635490 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.635494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.636360 (+ 866us) server_negotiation.cc:407] Connection header received
0926 14:49:11.637433 (+ 1073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.637436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.637510 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.637583 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.638749 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.639327 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.640444 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.640692 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.640782 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.643149 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.643152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.643162 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.643189 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.644494 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.644993 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.644996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.644998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.645071 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.645447 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.645451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.645453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.645643 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.645731 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.646316 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.646446 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 14:49:11.647485 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.635742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37787 (local address 127.0.0.1:56660)
0926 14:49:11.636198 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.636243 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:11.637227 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:11.637768 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:11.637775 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:11.638140 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:11.638608 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.638621 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.639452 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.639455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.640327 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.640336 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.640831 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.640835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.640996 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.641549 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:11.641565 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:11.642979 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:11.644629 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.644635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.644650 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.644900 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.645151 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.645154 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.645156 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.645355 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.645741 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:11.645747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:11.646072 (+ 325us) client_negotiation.cc:769] Sending connection context
0926 14:49:11.646249 (+ 177us) client_negotiation.cc:240] Negotiation successful
0926 14:49:11.646572 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":110,"threads_started":1}
I20260926 14:49:11.648808 23155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 14:49:11.657828 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } attempt: 3
I20260926 14:49:11.660678 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.649302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42721 (local address 127.0.0.1:36296)
0926 14:49:11.649401 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.649415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:11.649490 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:11.650219 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:11.650221 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:11.650361 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:11.650472 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.650477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.651904 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.651907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.652447 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.652455 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.653299 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.653304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.653438 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.654023 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:11.654039 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:11.656941 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:11.659116 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.659118 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.659120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.659344 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.659624 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.659628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.659630 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.659801 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.660206 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:11.660208 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:11.660303 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 14:49:11.660387 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 14:49:11.660490 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":23}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 14:49:11.661121 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.649372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.0.98.61:42721)
0926 14:49:11.649774 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.649786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.649811 (+ 25us) server_negotiation.cc:407] Connection header received
0926 14:49:11.649855 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.649861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.650063 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.650209 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.650582 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.651795 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.652599 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.653171 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.653288 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.657075 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.657085 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.657098 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.657121 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.659018 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.659440 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.659447 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.659449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.659549 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.659884 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.659886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.659888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.660121 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.660248 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.660499 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.660650 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":296,"thread_start_us":118,"threads_started":1}
I20260926 14:49:11.661708 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
/tmp/dist-test-taskR2lxu1/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33065
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37787,127.0.98.61:42721,127.0.98.60:33065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 14:49:11.663367 23155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 14:49:11.664120 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.663839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48272)
0926 14:49:11.663961 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.664049 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 14:49:11.664350 23105 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
I20260926 14:49:11.665248 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.665006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48276)
0926 14:49:11.665143 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.665187 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260926 14:49:11.665388 23155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
I20260926 14:49:11.667490 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.658369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36306 (local address 127.0.98.61:42721)
0926 14:49:11.658801 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.658809 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.658823 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:49:11.658856 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.658858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.658893 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.658962 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.659439 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.660150 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.661227 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.662109 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.664362 (+ 2253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.664486 (+ 124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.664501 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.664504 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.664522 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.666002 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.666386 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.666389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.666391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.666467 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.666707 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.666710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.666712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.666910 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.667000 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.667242 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.667342 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":168,"threads_started":1}
I20260926 14:49:11.667477 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.658326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42721 (local address 127.0.0.1:36306)
0926 14:49:11.658484 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.658500 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:11.658589 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:11.659023 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:11.659026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:11.659201 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:11.659332 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.659339 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.660290 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.660294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.661101 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.661114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.662251 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.662254 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.662369 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.662885 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:11.662899 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:11.664234 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:11.666104 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.666108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.666110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.666301 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.666547 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.666551 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.666552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.666633 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.666996 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:11.666999 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:11.667095 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 14:49:11.667183 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 14:49:11.667320 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 14:49:11.668165 23080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:49:11.668820 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.668605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48290)
0926 14:49:11.668717 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.668761 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 14:49:11.669480 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.669255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48296)
0926 14:49:11.669383 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.669427 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 14:49:11.669595 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
I20260926 14:49:11.700755 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } attempt: 1
I20260926 14:49:11.701650 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.701286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48306)
0926 14:49:11.701462 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.701552 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260926 14:49:11.702390 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.702170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48322)
0926 14:49:11.702277 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.702327 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 14:49:11.702522 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
I20260926 14:49:11.714648 23155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } attempt: 1
I20260926 14:49:11.715508 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.715218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48336)
0926 14:49:11.715366 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.715427 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 14:49:11.716159 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.715963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48350)
0926 14:49:11.716062 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.716105 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 14:49:11.716308 23155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
W20260926 14:49:11.772864 23162 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 14:49:11.773203 23162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 14:49:11.773248 23162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 14:49:11.773310 23162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 14:49:11.777649 23162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 14:49:11.777720 23162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 14:49:11.777750 23162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 14:49:11.777770 23162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 14:49:11.777801 23162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 14:49:11.777691 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } attempt: 2
I20260926 14:49:11.778548 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.778245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48356)
0926 14:49:11.778397 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.778462 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 14:49:11.779382 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.779142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48360)
0926 14:49:11.779269 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.779319 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 14:49:11.779500 23080 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
I20260926 14:49:11.783775 23162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37787,127.0.98.61:42721,127.0.98.60:33065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 14:04:34 UTC on ccycloud.quasar-ydwdgk.root.comops.site
build id 4674335
I20260926 14:49:11.784476 23155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } attempt: 2
I20260926 14:49:11.785064 23162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 14:49:11.785259 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.784965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48374)
0926 14:49:11.785110 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.785176 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 14:49:11.785925 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.785718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48390)
0926 14:49:11.785823 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.785872 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 14:49:11.786065 23155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33065: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33065: connect: Connection refused (error 111)
I20260926 14:49:11.786367 23162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 14:49:11.794608 23169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:11.794608 23168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 14:49:11.794623 23171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 14:49:11.794909 23162 server_base.cc:1034] running on GCE node
I20260926 14:49:11.795605 23162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 14:49:11.796754 23162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 14:49:11.797906 23162 hybrid_clock.cc:648] HybridClock initialized: now 1790434151797882 us; error 45 us; skew 500 ppm
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 14:49:11.800733 23162 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 14:49:11.801733 23162 webserver.cc:460] Webserver started at http://127.0.98.60:45621/ using document root <none> and password file <none>
I20260926 14:49:11.802321 23162 fs_manager.cc:359] Metadata directory not provided
I20260926 14:49:11.802388 23162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 14:49:11.802593 23162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 14:49:11.804723 23162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/data/instance:
uuid: "89bf05b83a2047f9a18c1b3f036244f0"
format_stamp: "Formatted at 2026-09-26 14:49:11 on dist-test-slave-s7k6"
server_key: "6f717aa094467516d00771c64580fa04"
server_key_iv: "03b011e2a829ec70152ce5f896e06d01"
server_key_version: "encryptionkey@0"
I20260926 14:49:11.805248 23162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/wal/instance:
uuid: "89bf05b83a2047f9a18c1b3f036244f0"
format_stamp: "Formatted at 2026-09-26 14:49:11 on dist-test-slave-s7k6"
server_key: "6f717aa094467516d00771c64580fa04"
server_key_iv: "03b011e2a829ec70152ce5f896e06d01"
server_key_version: "encryptionkey@0"
I20260926 14:49:11.809195 23162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 14:49:11.811872 23178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 14:49:11.813275 23162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 14:49:11.813393 23162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/wal
uuid: "89bf05b83a2047f9a18c1b3f036244f0"
format_stamp: "Formatted at 2026-09-26 14:49:11 on dist-test-slave-s7k6"
server_key: "6f717aa094467516d00771c64580fa04"
server_key_iv: "03b011e2a829ec70152ce5f896e06d01"
server_key_version: "encryptionkey@0"
I20260926 14:49:11.813541 23162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 14:49:11.836660 23162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 14:49:11.840212 23162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 14:49:11.840389 23162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 14:49:11.847359 23162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33065
I20260926 14:49:11.847661 23230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33065 every 8 connection(s)
I20260926 14:49:11.848387 23162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/data/info.pb
I20260926 14:49:11.851294 23231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 14:49:11.855262 23231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } has no permanent_uuid. Determining permanent_uuid...
I20260926 14:49:11.858597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu as pid 23162
I20260926 14:49:11.858743 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskR2lxu1/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790433872429009-392-0/minicluster-data/master-2/wal/instance
I20260926 14:49:11.859057 392 external_mini_cluster.cc:1428] Setting key 455b508abe6c5f3cfa2d5bec6faad02e
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 14:49:11.869706 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.857333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.0.98.62:37787)
0926 14:49:11.857520 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.857526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.858447 (+ 921us) server_negotiation.cc:407] Connection header received
0926 14:49:11.859493 (+ 1046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.859497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.859553 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.859618 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.861092 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.861782 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.863167 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.863386 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.863492 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.865982 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.865986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.865988 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.866020 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.868003 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.868429 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.868432 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.868433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.868508 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.868778 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.868781 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.868783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.868963 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.869044 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.869418 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.869540 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 14:49:11.870569 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.859694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.0.98.60:33065)
0926 14:49:11.860030 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.860039 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.860061 (+ 22us) server_negotiation.cc:407] Connection header received
0926 14:49:11.860183 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.860187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.860353 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.860526 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.861338 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.862381 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.863471 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.863855 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.863942 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.866428 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.866433 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.866443 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.866464 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.867930 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.868324 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.868329 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.868330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.868445 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.868695 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.868698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.868700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.869073 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.869265 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.869473 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.869664 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":137,"threads_started":1}
I20260926 14:49:11.870568 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.857738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37787 (local address 127.0.0.1:56662)
0926 14:49:11.858271 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.858322 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:11.859283 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:11.859793 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:11.859802 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:11.860287 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:11.860927 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.860949 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.861901 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.861904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.863055 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.863065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.863534 (+ 469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.863537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.863697 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.864260 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:11.864277 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:11.865839 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:11.868104 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.868112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.868123 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.868338 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.868587 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.868590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.868592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.868697 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.869057 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:11.869065 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:11.869194 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 14:49:11.869347 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 14:49:11.869847 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":266,"thread_start_us":132,"threads_started":1}
I20260926 14:49:11.871855 23231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 14:49:11.877204 23155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } attempt: 3
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 14:49:11.878645 23080 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } attempt: 3
I20260926 14:49:11.880836 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.872299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42721 (local address 127.0.0.1:36318)
0926 14:49:11.872406 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.872421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:11.872492 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:11.872761 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:11.872763 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:11.872922 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:11.873029 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.873034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.873935 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.873937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.874781 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.874791 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.875354 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.875357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.875436 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.875922 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:11.875936 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:11.877278 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:11.879426 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.879429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.879431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.879613 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.879870 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.879873 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.879875 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.879969 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.880366 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:11.880369 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:11.880461 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 14:49:11.880555 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 14:49:11.880676 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 14:49:11.880847 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.872408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36318 (local address 127.0.98.61:42721)
0926 14:49:11.872544 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.872548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.872562 (+ 14us) server_negotiation.cc:407] Connection header received
0926 14:49:11.872608 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.872611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.872659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.872736 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.873161 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.873825 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.874873 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.875235 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.875356 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.877774 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.877778 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.877780 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.877802 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.879346 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.879695 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.879698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.879700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.879774 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.880064 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.880067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.880069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.880276 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.880369 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.880610 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.880715 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 14:49:11.881456 23231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 14:49:11.885593 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.873167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56668 (local address 127.0.98.62:37787)
0926 14:49:11.873280 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.873284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.873295 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:49:11.873348 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.873350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.873400 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.873469 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.874004 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.874740 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.875752 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.876073 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.876175 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.878601 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.878605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.878607 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.878880 (+ 273us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.883367 (+ 4487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.883866 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.883869 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.883871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.883952 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.884274 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.884277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.884279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.884483 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.885218 (+ 735us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.885299 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.885437 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 14:49:11.886075 23157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.877613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48410)
0926 14:49:11.877711 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.877724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:11.877794 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:11.878018 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:11.878022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:11.878189 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:11.878292 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.878300 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.879286 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.879289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.879992 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.880001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.880631 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.880634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.880716 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.881114 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:11.881130 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:11.882502 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:11.884302 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.884304 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.884307 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.884550 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.884812 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.884815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.884817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.884897 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.885314 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:11.885317 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:11.885406 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 14:49:11.885508 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 14:49:11.885924 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 14:49:11.886610 23235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.877677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.0.98.60:33065)
0926 14:49:11.877787 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.877791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.877804 (+ 13us) server_negotiation.cc:407] Connection header received
0926 14:49:11.877866 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.877869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.877912 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.877988 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.878409 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.879187 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.880116 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.880443 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.880526 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.882626 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.882630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.882632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.882654 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.884186 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.884641 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.884644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.884646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.884721 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.884984 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.884987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.884989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.885157 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.885231 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.885888 (+ 657us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.886468 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":444,"server-negotiator.queue_time_us":49}
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 14:49:11.890285 23155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9: Bootstrap starting.
I20260926 14:49:11.890714 23083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.879026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48416)
0926 14:49:11.879336 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.879349 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:11.879414 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:11.880157 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:11.880160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:11.880302 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:11.880390 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.880396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.881736 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.881740 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.882473 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.882485 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.883656 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.883660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.883751 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.884152 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:11.884165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:11.885547 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:11.888512 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.888516 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.888518 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.888738 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.889015 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.889019 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.889021 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.889337 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.889999 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:11.890002 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:11.890098 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 14:49:11.890178 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 14:49:11.890309 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260926 14:49:11.891342 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.879649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.0.98.60:33065)
0926 14:49:11.879969 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.879978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.879989 (+ 11us) server_negotiation.cc:407] Connection header received
0926 14:49:11.880016 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.880019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.880057 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.880127 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.880478 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.881632 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.882644 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.883262 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.885654 (+ 2392us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.886043 (+ 389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.886047 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.886049 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.886067 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.888403 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.888831 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.888834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.888836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.888913 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.889472 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.889475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.889477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.889887 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.890987 (+ 1100us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.891061 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.891170 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":258,"spinlock_wait_cycles":5376,"thread_start_us":103,"threads_started":1}
I20260926 14:49:11.891904 23233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.881867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33065 (local address 127.0.0.1:48432)
0926 14:49:11.882666 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 14:49:11.882678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 14:49:11.882740 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 14:49:11.883021 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 14:49:11.883024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 14:49:11.883485 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 14:49:11.883559 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.883566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.884512 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.884515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.885212 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 14:49:11.885219 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.885797 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.885800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 14:49:11.885864 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.886817 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 14:49:11.886833 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 14:49:11.888725 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 14:49:11.890516 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.890520 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.890522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.890748 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.891050 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 14:49:11.891053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 14:49:11.891055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 14:49:11.891126 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 14:49:11.891497 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 14:49:11.891500 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 14:49:11.891574 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 14:49:11.891653 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 14:49:11.891764 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":144}
I20260926 14:49:11.892373 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.881946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.0.98.60:33065)
0926 14:49:11.882196 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.882203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.882829 (+ 626us) server_negotiation.cc:407] Connection header received
0926 14:49:11.882853 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.882856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.882893 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.882969 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.883668 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.884415 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.885305 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.885675 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.885760 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.888844 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.888847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.888850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.888867 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.890408 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.890872 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.890876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.890877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.890968 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.891209 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.891212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.891214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.891382 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.891465 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.892124 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.892227 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":169,"server-negotiator.queue_time_us":193,"thread_start_us":106,"threads_started":1}
Sep 26 14:49:11 dist-test-slave-s7k6 krb5kdc[22999](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434151, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 14:49:11.893386 23155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:11.894308 23155 log.cc:824] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:11.894383 23080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55: Bootstrap starting.
I20260926 14:49:11.896723 23155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9: No bootstrap required, opened a new log
I20260926 14:49:11.897262 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.886726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.0.98.61:42721)
0926 14:49:11.886849 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.886852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.886864 (+ 12us) server_negotiation.cc:407] Connection header received
0926 14:49:11.886973 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.886976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.887018 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.887082 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.888065 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.888691 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.889635 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.889957 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.890038 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.893344 (+ 3306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.893348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.893351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.893376 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.895023 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.895554 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.895558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.895560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.895643 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.895912 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.895915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.895917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.896128 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.896279 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.896987 (+ 708us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.897104 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 14:49:11.897353 23080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:11.897773 23231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0: Bootstrap starting.
I20260926 14:49:11.898254 23080 log.cc:824] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:11.900418 23231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0: Neither blocks nor log segments found. Creating new log.
I20260926 14:49:11.900523 23080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55: No bootstrap required, opened a new log
I20260926 14:49:11.900554 23155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } }
I20260926 14:49:11.900789 23155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af81368bcdc429a8724963af7c49cc9, State: Initialized, Role: FOLLOWER
I20260926 14:49:11.901110 23155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } }
I20260926 14:49:11.901288 23231 log.cc:824] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0: Log is configured to *not* fsync() on all Append() calls
I20260926 14:49:11.901576 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } } }
I20260926 14:49:11.901783 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:11.902458 23155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:11.904000 23231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0: No bootstrap required, opened a new log
I20260926 14:49:11.904942 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 14:49:11.905592 23080 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } }
I20260926 14:49:11.905800 23080 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3c80c6d91bc4ef1ac5fb59b2e55db55, State: Initialized, Role: FOLLOWER
I20260926 14:49:11.906131 23080 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } }
I20260926 14:49:11.906212 23054 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56668
I20260926 14:49:11.907524 23080 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:11.907620 23054 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 14:49:11.907737 23054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:49:11.908731 23231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } }
I20260926 14:49:11.908987 23231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89bf05b83a2047f9a18c1b3f036244f0, State: Initialized, Role: FOLLOWER
I20260926 14:49:11.909008 23054 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } }
I20260926 14:49:11.906176 23087 master_runner.cc:186] Error getting master registration for 127.0.98.62:37787: OK, instance_id { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" instance_seqno: 1790434151415720 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 14:49:11.909327 23231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } }
I20260926 14:49:11.909487 23087 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:49:11.909947 23054 leader_election.cc:288] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [CANDIDATE]: Term 1 election: Requested vote from peers 9af81368bcdc429a8724963af7c49cc9 (127.0.98.61:42721), 89bf05b83a2047f9a18c1b3f036244f0 (127.0.98.60:33065)
I20260926 14:49:11.910627 23231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 14:49:11.911238 23129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9af81368bcdc429a8724963af7c49cc9"
I20260926 14:49:11.911473 23129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 14:49:11.912727 23129 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c80c6d91bc4ef1ac5fb59b2e55db55 in term 1.
I20260926 14:49:11.912899 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } } }
I20260926 14:49:11.913007 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:11.913182 23205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "89bf05b83a2047f9a18c1b3f036244f0"
I20260926 14:49:11.913303 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } } }
I20260926 14:49:11.913417 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:11.913380 23205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [term 0 FOLLOWER]: Advancing to term 1
W20260926 14:49:11.913486 23033 tablet.cc:2369] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 14:49:11.913918 23031 leader_election.cc:302] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9af81368bcdc429a8724963af7c49cc9, f3c80c6d91bc4ef1ac5fb59b2e55db55; no voters:
I20260926 14:49:11.914693 23205 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3c80c6d91bc4ef1ac5fb59b2e55db55 in term 1.
I20260926 14:49:11.915930 23253 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 14:49:11.916124 23253 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [term 1 LEADER]: Becoming Leader. State: Replica: f3c80c6d91bc4ef1ac5fb59b2e55db55, State: Running, Role: LEADER
W20260926 14:49:11.916366 23264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:49:11.916441 23264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 14:49:11.916493 23277 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 14:49:11.916477 23253 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } }
W20260926 14:49:11.916555 23277 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:49:11.918512 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3c80c6d91bc4ef1ac5fb59b2e55db55. Latest consensus state: current_term: 1 leader_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } } }
I20260926 14:49:11.918612 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:11.919595 23012 master_runner.cc:263] No leader master found from master f3c80c6d91bc4ef1ac5fb59b2e55db55
I20260926 14:49:11.919656 23012 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:49:11.919699 23285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 14:49:11.920192 23162 master_runner.cc:263] No leader master found from master 89bf05b83a2047f9a18c1b3f036244f0
I20260926 14:49:11.920243 23162 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 14:49:11.920612 23285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260926 14:49:11.923233 23252 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 14:49:11.924625 23252 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 14:49:11.930420 23205 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [term 1 FOLLOWER]: Refusing update from remote peer f3c80c6d91bc4ef1ac5fb59b2e55db55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:49:11.931102 23283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:49:11.932464 23129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [term 1 FOLLOWER]: Refusing update from remote peer f3c80c6d91bc4ef1ac5fb59b2e55db55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 14:49:11.934448 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.923724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36342 (local address 127.0.98.61:42721)
0926 14:49:11.923840 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.923844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.924370 (+ 526us) server_negotiation.cc:407] Connection header received
0926 14:49:11.924825 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.924829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.924876 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.924941 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.925944 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.926743 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.927898 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.928223 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.928297 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.930405 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.930409 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.930413 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.930451 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.932454 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.933038 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.933041 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.933043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.933132 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.933453 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.933457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.933459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.933682 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.933783 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.934144 (+ 361us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.934274 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 14:49:11.936480 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3c80c6d91bc4ef1ac5fb59b2e55db55. Latest consensus state: current_term: 1 leader_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } } }
I20260926 14:49:11.936607 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:11.936720 23283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 14:49:11.941097 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3c80c6d91bc4ef1ac5fb59b2e55db55. Latest consensus state: current_term: 1 leader_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } } }
I20260926 14:49:11.941238 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:11.945586 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } } }
I20260926 14:49:11.945739 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:11.947041 23265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } } }
I20260926 14:49:11.947170 23265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:11.947386 23084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.925024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.0.98.62:37787)
0926 14:49:11.925131 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.925135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.936378 (+ 11243us) server_negotiation.cc:407] Connection header received
0926 14:49:11.936568 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.936572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.936626 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.936691 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.937677 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.938463 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.939604 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.939937 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.940011 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.941827 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.941831 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.941834 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.941861 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.944493 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.945930 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.945935 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.945937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.946038 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.946370 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.946373 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.946375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.946601 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.946704 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.947089 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.947217 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 14:49:11.947578 23287 mvcc.cc:205] Tried to move back new op lower bound from 7333618286299197440 to 7333618286253395968. Current Snapshot: MvccSnapshot[applied={T|T < 7333618286299197440}]
I20260926 14:49:11.950148 23253 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } } }
I20260926 14:49:11.950268 23253 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55 [sys.catalog]: This master's current role is: LEADER
I20260926 14:49:11.951124 23240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3c80c6d91bc4ef1ac5fb59b2e55db55" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37787 } } peers { permanent_uuid: "9af81368bcdc429a8724963af7c49cc9" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42721 } } peers { permanent_uuid: "89bf05b83a2047f9a18c1b3f036244f0" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33065 } } }
I20260926 14:49:11.951246 23240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 14:49:11.952667 23293 mvcc.cc:205] Tried to move back new op lower bound from 7333618286299197440 to 7333618286253395968. Current Snapshot: MvccSnapshot[applied={T|T < 7333618286299197440}]
I20260926 14:49:11.955197 23285 catalog_manager.cc:1347] Generated new cluster ID: 2644f61824e64c65b3a759d355fb1372
I20260926 14:49:11.955281 23285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 14:49:11.956535 23236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:11.924514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.0.98.60:33065)
0926 14:49:11.924618 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:11.924622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:11.929282 (+ 4660us) server_negotiation.cc:407] Connection header received
0926 14:49:11.941478 (+ 12196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:11.941481 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:11.941540 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:11.941591 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:11.942371 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.943212 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.948218 (+ 5006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:11.951134 (+ 2916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:11.951239 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:11.952955 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 14:49:11.952959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 14:49:11.952962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 14:49:11.952990 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 14:49:11.954602 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.955095 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.955099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.955101 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.955181 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 14:49:11.955587 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 14:49:11.955591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 14:49:11.955593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 14:49:11.955802 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 14:49:11.955914 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:11.956223 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 14:49:11.956361 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 14:49:12.016641 23285 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 14:49:12.018275 23285 catalog_manager.cc:1504] Loading token signing keys...
I20260926 14:49:12.044453 23285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3c80c6d91bc4ef1ac5fb59b2e55db55: Generated new TSK 0
I20260926 14:49:12.045424 23285 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 14:49:12.081144 23161 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.077752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36354 (local address 127.0.98.61:42721)
0926 14:49:12.077904 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.077909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.079997 (+ 2088us) server_negotiation.cc:407] Connection header received
0926 14:49:12.080211 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.080215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.080269 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.080322 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.081003 (+ 681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36354: BlockingRecv error: recv got EOF from 127.0.0.1:36354 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 14:49:12.082630 23236 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.078624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48446 (local address 127.0.98.60:33065)
0926 14:49:12.078796 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.078801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.081677 (+ 2876us) server_negotiation.cc:407] Connection header received
0926 14:49:12.081790 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.081793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.081849 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.081939 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.082512 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48446: BlockingRecv error: recv got EOF from 127.0.0.1:48446 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 14:49:12.083067 23084 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.079327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56690 (local address 127.0.98.62:37787)
0926 14:49:12.079452 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.079457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.082055 (+ 2598us) server_negotiation.cc:407] Connection header received
0926 14:49:12.082242 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.082246 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.082299 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.082353 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.082946 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56690: BlockingRecv error: recv got EOF from 127.0.0.1:56690 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260926 14:49:12.087608 23161 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.085569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36370 (local address 127.0.98.61:42721)
0926 14:49:12.085686 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.085690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.086548 (+ 858us) server_negotiation.cc:407] Connection header received
0926 14:49:12.086800 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.086804 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.086897 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.086962 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.087499 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36370: BlockingRecv error: recv got EOF from 127.0.0.1:36370 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 14:49:12.092571 23161 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.091086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36372 (local address 127.0.98.61:42721)
0926 14:49:12.091187 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.091191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.091703 (+ 512us) server_negotiation.cc:407] Connection header received
0926 14:49:12.091885 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.091888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.091936 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.091982 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.092461 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36372: BlockingRecv error: recv got EOF from 127.0.0.1:36372 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 14:49:12.098524 23161 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.096789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36384 (local address 127.0.98.61:42721)
0926 14:49:12.096917 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.096922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.097540 (+ 618us) server_negotiation.cc:407] Connection header received
0926 14:49:12.097745 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.097749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.097801 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.097865 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.098401 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36384: BlockingRecv error: recv got EOF from 127.0.0.1:36384 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 14:49:12.106405 23161 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.104660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36388 (local address 127.0.98.61:42721)
0926 14:49:12.104795 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.104799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.105422 (+ 623us) server_negotiation.cc:407] Connection header received
0926 14:49:12.105636 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.105640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.105694 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.105745 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.106279 (+ 534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36388: BlockingRecv error: recv got EOF from 127.0.0.1:36388 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260926 14:49:12.118105 23161 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.116753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.0.98.61:42721)
0926 14:49:12.116870 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.116874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.117247 (+ 373us) server_negotiation.cc:407] Connection header received
0926 14:49:12.117366 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.117369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.117426 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.117502 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.117967 (+ 465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36396: BlockingRecv error: recv got EOF from 127.0.0.1:36396 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 14:49:12.137305 23161 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.136015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.0.98.61:42721)
0926 14:49:12.136134 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.136138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.136423 (+ 285us) server_negotiation.cc:407] Connection header received
0926 14:49:12.136585 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.136589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.136640 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.136707 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.137168 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36402: BlockingRecv error: recv got EOF from 127.0.0.1:36402 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 14:49:12.172370 23161 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.171205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.0.98.61:42721)
0926 14:49:12.171357 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.171360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.171451 (+ 91us) server_negotiation.cc:407] Connection header received
0926 14:49:12.171625 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.171628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.171682 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.171768 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.172263 (+ 495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36410: BlockingRecv error: recv got EOF from 127.0.0.1:36410 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260926 14:49:12.239678 23161 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.238575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36420 (local address 127.0.98.61:42721)
0926 14:49:12.238725 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.238729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.238823 (+ 94us) server_negotiation.cc:407] Connection header received
0926 14:49:12.238978 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.238981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.239031 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.239101 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.239557 (+ 456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36420: BlockingRecv error: recv got EOF from 127.0.0.1:36420 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 14:49:12.370810 23161 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.369695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36430 (local address 127.0.98.61:42721)
0926 14:49:12.369883 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.369887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.369945 (+ 58us) server_negotiation.cc:407] Connection header received
0926 14:49:12.370071 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.370073 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.370121 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.370192 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.370630 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36430: BlockingRecv error: recv got EOF from 127.0.0.1:36430 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260926 14:49:12.629935 23161 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 14:49:12.628839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36442 (local address 127.0.98.61:42721)
0926 14:49:12.628954 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:12.628958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:12.629060 (+ 102us) server_negotiation.cc:407] Connection header received
0926 14:49:12.629187 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:12.629190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:12.629248 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:12.629346 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 14:49:12.629831 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36442: BlockingRecv error: recv got EOF from 127.0.0.1:36442 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 14:49:12.917882 23277 catalog_manager.cc:1259] Loaded cluster ID: 2644f61824e64c65b3a759d355fb1372
I20260926 14:49:12.917963 23277 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0: loading cluster ID for follower catalog manager: success
I20260926 14:49:12.919724 23277 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0: acquiring CA information for follower catalog manager: success
I20260926 14:49:12.920881 23277 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 89bf05b83a2047f9a18c1b3f036244f0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:49:12.925931 23252 catalog_manager.cc:1259] Loaded cluster ID: 2644f61824e64c65b3a759d355fb1372
I20260926 14:49:12.926005 23252 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9: loading cluster ID for follower catalog manager: success
I20260926 14:49:12.927682 23252 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9: acquiring CA information for follower catalog manager: success
I20260926 14:49:12.928841 23252 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9af81368bcdc429a8724963af7c49cc9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 14:49:13.148265 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:13.144194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36446 (local address 127.0.98.61:42721)
0926 14:49:13.144453 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:13.144458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:13.144473 (+ 15us) server_negotiation.cc:407] Connection header received
0926 14:49:13.144610 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:13.144612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:13.144742 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:13.144839 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:49:13.145108 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:13.145656 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:13.146547 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:13.146823 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:13.147029 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:13.147328 (+ 299us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:49:13.147812 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:49:13.147945 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:13.148043 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 14:49:13.148104 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":67,"threads_started":1}
I20260926 14:49:13.154824 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:13.151227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.0.98.60:33065)
0926 14:49:13.151462 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:13.151465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:13.151481 (+ 16us) server_negotiation.cc:407] Connection header received
0926 14:49:13.151560 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:13.151562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:13.151674 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:13.151768 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:49:13.152005 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:13.152526 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:13.153422 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:13.153655 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:13.153749 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:13.154020 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:49:13.154469 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:49:13.154566 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:13.154597 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 14:49:13.154649 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":62,"threads_started":1}
I20260926 14:49:13.162034 23369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:13.157976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.0.98.60:33065)
0926 14:49:13.158070 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:13.158074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:13.158311 (+ 237us) server_negotiation.cc:407] Connection header received
0926 14:49:13.158448 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:13.158451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:13.158493 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:13.158550 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:49:13.158896 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:13.159713 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:13.160938 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:13.161157 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:13.161227 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:13.161569 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:49:13.161646 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:49:13.161703 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:13.161830 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 14:49:13.161890 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 14:49:13.162294 23363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:13.157812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.0.98.61:42721)
0926 14:49:13.157957 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:13.157961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:13.158401 (+ 440us) server_negotiation.cc:407] Connection header received
0926 14:49:13.158587 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:13.158590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:13.158644 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:13.158706 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:49:13.159095 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:13.159592 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:13.160896 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:13.161346 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:13.161437 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:13.161806 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:49:13.161886 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:49:13.161962 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:13.162100 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 14:49:13.162164 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 14:49:13.162590 23375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 14:49:13.157831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.0.98.62:37787)
0926 14:49:13.158135 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0926 14:49:13.158139 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 14:49:13.158298 (+ 159us) server_negotiation.cc:407] Connection header received
0926 14:49:13.158408 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 14:49:13.158411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 14:49:13.158620 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 14:49:13.158730 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 14:49:13.159143 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:13.159922 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:13.160729 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 14:49:13.161101 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 14:49:13.161214 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 14:49:13.161529 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 14:49:13.161950 (+ 421us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 14:49:13.162038 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 14:49:13.162308 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 14:49:13.162444 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":66,"threads_started":1}
I20260926 14:49:13.165954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 23012
I20260926 14:49:13.171622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 23087
I20260926 14:49:13.176731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskR2lxu1/build/debug/bin/kudu with pid 23162
2026-09-26T14:49:13Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3014 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6075 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280739 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST